From patchwork Mon Apr 6 23:16:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11476835 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57ACA14DD for ; Mon, 6 Apr 2020 23:16:32 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2DC3B206C0 for ; Mon, 6 Apr 2020 23:16:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="eefZXReT"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="LwoWFesn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2DC3B206C0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jmOTBRQUnIxrsjjGwviHp57nWYSWgf0rfsP4qQUfSdo=; b=eefZXReTlbSD+g dg5hdGqe5sY2d6RXT4SPZAONJuhj0gFOyyApnVWpYrCTDUZYf1ri37BcBwZC51+6O2/SWqVqQcCWz MhVAHEv2G6Xjxn6D46ldET4UyzCcgD37zanAmyCkhkOQGzPnWzhvWovW6k8hBM0eA8wMTnmnQFP3C mASXBKFwf1yMsQeM+x6P7U2Bh64EQSOZyM802yztpKuh7imeqGDMxDgvSRZk+aOpZmmMcgV3iqlhz 4+DqXMQo0tX4+zEtMYP5ItCWnIQg+WCR/Vm7JwXclnwvJaNU6P7Isbc4X2zjxnmide6ihQ3RXN6GH 6MOE5n/FFCjFUG3mF8zA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayh-0003pn-37; Mon, 06 Apr 2020 23:16:27 +0000 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayT-0003d7-8P for linux-arm-kernel@lists.infradead.org; Mon, 06 Apr 2020 23:16:14 +0000 Received: by mail-pg1-x543.google.com with SMTP id m17so792017pgj.5 for ; Mon, 06 Apr 2020 16:16:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+xhCyqQB0W40O85FTKh55+NUFCgAzlVgGvF/KPoh1Gc=; b=LwoWFesn1H5UXh6fxlgOTqCRawwYzGBkOsIJMdXLsW9VNmWZh7FlN5g0w5ME0mavLf En4DY8Qr/H4pZXCOY4PL6HPF/8zw4SdQLpp2BhqVIanjk6OLqEW7lqoonhxVawZOThPI RmDZ3S1j9plwAD/vf8OX3Fgc4njkJ0a8Khu3A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+xhCyqQB0W40O85FTKh55+NUFCgAzlVgGvF/KPoh1Gc=; b=ImvMKqg27vnx+DhC9EwOcxXF3ZdTf8079nDbHH3wCjhoYNHQ2POXOKpdm4bVBqEZev 3DxkWaiKucNCQCPXOMhy7qwbRghM3JAaeR+XQPeQ3sqnKrvKGVA2tlGSv02S48aEhF3R vJL5wYzhSfdo7RI020R3IV44Kq7S0jP4qKd4fbY3B/4Qa4PnLQ55DK3OGSE9zNCYwJ0r 05BRuVhaawo6VGZtiEd2YHXJhbd3HnggsGOH/Y3I8t7rbhGrpFmuRuqNDdzSaAmNvB6o U53OrGl5Mc4DLWOcd7TxOutoRCFSvD7/ljxjo2A88HcMxPEvzpQ5mNWDSg3Q4snsd7+d 6JxA== X-Gm-Message-State: AGi0PuZkG5OVU5/8j02r3dwNSd242QnOyEgU96zpdQYWj7hB0TDwQIya fJQqLpf9CPpclQZ4GPzfiNSyGQ== X-Google-Smtp-Source: APiQypIdiCJFOXl5opnnyyI8GIGFkvdvihjT0dOslocyOZv32kqWFFk5saHzi7q9MToGH7pRu5xnlA== X-Received: by 2002:a63:7b1d:: with SMTP id w29mr1421905pgc.4.1586214972791; Mon, 06 Apr 2020 16:16:12 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id c190sm12363077pfa.66.2020.04.06.16.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Apr 2020 16:16:10 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v3 1/5] jump_label: Provide CONFIG-driven build state defaults Date: Mon, 6 Apr 2020 16:16:02 -0700 Message-Id: <20200406231606.37619-2-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200406231606.37619-1-keescook@chromium.org> References: <20200406231606.37619-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200406_161613_302109_2A43869D X-CRM114-Status: GOOD ( 13.90 ) X-Spam-Score: -0.4 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:543 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.2 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Choosing the initial state of static branches changes the assembly layout (if the condition is expected to be likely, inline, or unlikely, out of line via a jump). A few places in the kernel use (or could be using) a CONFIG to choose the default state, so provide the infrastructure to do this and convert the existing cases (init_on_alloc and init_on_free) to the new macros. Signed-off-by: Kees Cook Acked-by: Peter Zijlstra (Intel) --- include/linux/jump_label.h | 19 +++++++++++++++++++ include/linux/mm.h | 12 ++---------- mm/page_alloc.c | 12 ++---------- 3 files changed, 23 insertions(+), 20 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 3526c0aee954..615fdfb871a3 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -382,6 +382,21 @@ struct static_key_false { [0 ... (count) - 1] = STATIC_KEY_FALSE_INIT, \ } +#define _DEFINE_STATIC_KEY_1(name) DEFINE_STATIC_KEY_TRUE(name) +#define _DEFINE_STATIC_KEY_0(name) DEFINE_STATIC_KEY_FALSE(name) +#define DEFINE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_, IS_ENABLED(cfg))(name) + +#define _DEFINE_STATIC_KEY_RO_1(name) DEFINE_STATIC_KEY_TRUE_RO(name) +#define _DEFINE_STATIC_KEY_RO_0(name) DEFINE_STATIC_KEY_FALSE_RO(name) +#define DEFINE_STATIC_KEY_MAYBE_RO(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_RO_, IS_ENABLED(cfg))(name) + +#define _DECLARE_STATIC_KEY_1(name) DECLARE_STATIC_KEY_TRUE(name) +#define _DECLARE_STATIC_KEY_0(name) DECLARE_STATIC_KEY_FALSE(name) +#define DECLARE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DECLARE_STATIC_KEY_, IS_ENABLED(cfg))(name) + extern bool ____wrong_branch_error(void); #define static_key_enabled(x) \ @@ -482,6 +497,10 @@ extern bool ____wrong_branch_error(void); #endif /* CONFIG_JUMP_LABEL */ +#define static_branch_maybe(config, x) \ + (IS_ENABLED(config) ? static_branch_likely(x) \ + : static_branch_unlikely(x)) + /* * Advanced usage; refcount, branch is enabled when: count != 0 */ diff --git a/include/linux/mm.h b/include/linux/mm.h index c54fb96cb1e6..059658604dd6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2662,11 +2662,7 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_alloc); -#else -DECLARE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc) && @@ -2675,11 +2671,7 @@ static inline bool want_init_on_alloc(gfp_t flags) return flags & __GFP_ZERO; } -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_free); -#else -DECLARE_STATIC_KEY_FALSE(init_on_free); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { return static_branch_unlikely(&init_on_free) && diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3c4eb750a199..1f625e5a03c0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -135,18 +135,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_alloc); -#else -DEFINE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_free); -#else -DEFINE_STATIC_KEY_FALSE(init_on_free); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static int __init early_init_on_alloc(char *buf) From patchwork Mon Apr 6 23:16:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11476823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A5CB1805 for ; Mon, 6 Apr 2020 23:16:17 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 688BF206C0 for ; Mon, 6 Apr 2020 23:16:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Wki1uy6E"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DETEeYLU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 688BF206C0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DYOuuls3vUmYPV/yP1ZN24o59puDJn5BRzUCTEzvfuc=; b=Wki1uy6EvKqvyV FR6aG1rbG38oPZr4idtqv2dAzDXBK+TV1PgE5bkhW+v98TND3Xij5UZmUWkfmlmTdqIvDW+ImM739 KiUpqu0mHaSaGzIdK+1Nddd+QPyfR3l6L216oRLa0MN0XIlLSzgiDekGUrtNLyki3vWt5fR9nvonX z7OzruFBN9JZUCFrIFxlRVzj89sCSb/rnAibez6MyXsOiZpQ+CuFiAEDg1pKd2/celi5NJdnFmypk p0K+L+mMqHuDB70igJq+5xn5xlphGnAdCb3lh5QLqQWOglIptzuSk55DWRV6hRyDwg4a4TA7jvSMA hIx+UnqSMukKdJQyW5bA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayW-0003eQ-01; Mon, 06 Apr 2020 23:16:16 +0000 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayT-0003cy-1S for linux-arm-kernel@lists.infradead.org; Mon, 06 Apr 2020 23:16:14 +0000 Received: by mail-pg1-x543.google.com with SMTP id m13so785405pgd.8 for ; Mon, 06 Apr 2020 16:16:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jjDe4Hi7TtVO/wKC4Ixr8yGihHqa400mWL0wehOCRBk=; b=DETEeYLUPrHDWmX+mgf3XfkHhQ8bXZ4JQ20xi9rnXY3vkREikPg39QMkW+FwOTZOB6 ad3hnyhmQW1o2fY8ksqfVaf3PHYZJN+QwgTMVfcpg7yFW8qoLnAx5xje3nNlmfgKQLKi xbiidiGJHcDcOJhrAq91ofcxuchLHrHgbpU+M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jjDe4Hi7TtVO/wKC4Ixr8yGihHqa400mWL0wehOCRBk=; b=Q0HR1Q57LjgGXAvEgVMudccvs1mbAZA7/8/I3gPLb7ZryicNxzshNeCxo+sJV1uxig DZneE0hh10KCX+SpqfSQ3iw0MELwS++gfrE2xMXeWDcZ0e2Zxvkp4Z6fS25TctmfTTAb zeImiILwmgI2YbN0xfaIbsaNlV9gxctkAZ7n7/a93rJErbk1MGOQlm6TpPRvmENcey1c /DoF8c3Qtfcf8xVOUxgUOcxCGb35JhCqxDVp3suxATIPxqsql3EqIqrhc0ry1+p1QMhy AFQRFSoJjshgZDdWL6ZdpGSrwTHLFk+NYtG4VZxoCj2599bWfDxICrMNSUcFnhefr0tD oeKw== X-Gm-Message-State: AGi0PuYye54KELmlQe0wFgc08807t2qkjZOig0/tvtfltab81Q5QwlPW 9eHRFhwFTf54Vrphl1SI8uEutA== X-Google-Smtp-Source: APiQypJOAJUD5o/cOCh1/7iAsdO0knHWBcNha6PAtRGiz4cvm/tS0aBBSt7jxUfJb4cvKE69NcjkTA== X-Received: by 2002:a63:7b5e:: with SMTP id k30mr1393245pgn.209.1586214971506; Mon, 06 Apr 2020 16:16:11 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id 74sm12190832pfy.120.2020.04.06.16.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Apr 2020 16:16:10 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v3 2/5] init_on_alloc: Unpessimize default-on builds Date: Mon, 6 Apr 2020 16:16:03 -0700 Message-Id: <20200406231606.37619-3-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200406231606.37619-1-keescook@chromium.org> References: <20200406231606.37619-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200406_161613_106373_A37CFFC8 X-CRM114-Status: GOOD ( 12.88 ) X-Spam-Score: -0.4 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:543 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.2 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Right now, the state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branch tests. Use the new jump_label macro to check CONFIG settings to default to the "expected" state, unpessimizes the resulting assembly code. Signed-off-by: Kees Cook Reviewed-by: Alexander Potapenko --- include/linux/mm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 059658604dd6..64e911159ffa 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2665,7 +2665,8 @@ static inline void kernel_poison_pages(struct page *page, int numpages, DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc) && + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc) && !page_poisoning_enabled()) return true; return flags & __GFP_ZERO; @@ -2674,7 +2675,8 @@ static inline bool want_init_on_alloc(gfp_t flags) DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free) && + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free) && !page_poisoning_enabled(); } From patchwork Mon Apr 6 23:16:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11476851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F39E14DD for ; Mon, 6 Apr 2020 23:17:08 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 77E18206C3 for ; Mon, 6 Apr 2020 23:17:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="H2fyzj5b"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="jFwMyhOo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 77E18206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=u7c/E3jbfel8TkytpYzJU+/6RuctWpBWFvBI9Z2MhAQ=; b=H2fyzj5b9KUhnV X9P2jzFTDQVo8kXAN6/KFAsNCsGdYl/vK4kKkoSKhMnBX1PvIm3JzvKDVyZAEmCiJOUbu43Ab1QMo BXjGbZVVjmIEQwE7xQIZo+7KbAPGO+b4+tfAzHg7dT2l+4leucq/HOPkCT/1yX6ZQDoDMQtg3eq65 PownZvKlPSKrX1qoCUrR5QXIfyQV/SbQHOsKb3vCTq7KUc5L8y+kM5lObK7nhbsooK7dlacLMBV1d MB5Lm8JFOAhIgIsO53ePXwCSsEvwPcfP8seKyz3QUcLvsph+UyAS/VL6dOpNU3XEf23Y4HqQd9YOG 94ds+2l/NvNBk0j58NfQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLazK-0004XK-7z; Mon, 06 Apr 2020 23:17:06 +0000 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayU-0003dl-Dt for linux-arm-kernel@lists.infradead.org; Mon, 06 Apr 2020 23:16:16 +0000 Received: by mail-pj1-x1041.google.com with SMTP id ng8so565840pjb.2 for ; Mon, 06 Apr 2020 16:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gI5IFR42YfojEVReCokIL0PkImea/rfn/M9y/tkHY3U=; b=jFwMyhOoOhIidIm2MRbLuwmjjsTl0duo2bGeUwCoUDkarOJ9jJsT13+uJWwEt5weB7 uBqhEnMU5ufwmm6gsM6owJ74S1MB1iiKDZl4BywfhE04wcjYNTqGdNXMKBkwSCibT2Q2 Ppmgzkw/R5mxGJHXnmX3KKpxpDz0ThfqUGMAE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gI5IFR42YfojEVReCokIL0PkImea/rfn/M9y/tkHY3U=; b=GKJlNH/XwavSbm+R3uKo4zPybyieZUQ1YKO256OWzfTdDRai5HSAlpd0fUovRP1aTe W7gb/bqH3fVBOBcjcU2Bi2sTAZ1zS9pVhREFy2dfl61/NUS3c0Dv7jdaUJfL8HXnIBM2 H2atYL7b63oDMcx/eEHk75kChe65KSo08Qp6zLX0tTvgXZSK6WVfhYfykmgsudDiLf+I 8xxumIrhy2+VrSHvY20LNcT+q69vC73pBJpB2yYFYM8Jml+0QuWK0v0ZiLrkB0V20rBw ZpvNCt6a86ICFP/G81yf4pUOVX8P1Kxq5tucQaFsfBm7L5reh7MeVcJBARRZ+OnTOEJl 0fhA== X-Gm-Message-State: AGi0PubycVIIA7unn6Q8X8wus4KK2yXEqNV2caN81Og8GcnBkTfz8Eq7 /p8i7YZTgazhdXx0/ArAoiLeNw== X-Google-Smtp-Source: APiQypLr0HSVNkvDvGSyptY8A0rxNb1GV4o+xBzFEZBbbppMCeIsWmsRLIX7ABdxlixe031V5gjLuA== X-Received: by 2002:a17:902:8c94:: with SMTP id t20mr21940332plo.336.1586214973377; Mon, 06 Apr 2020 16:16:13 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id t27sm3727621pgn.53.2020.04.06.16.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Apr 2020 16:16:10 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v3 3/5] stack: Optionally randomize kernel stack offset each syscall Date: Mon, 6 Apr 2020 16:16:04 -0700 Message-Id: <20200406231606.37619-4-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200406231606.37619-1-keescook@chromium.org> References: <20200406231606.37619-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200406_161614_477104_514CA65F X-CRM114-Status: GOOD ( 29.91 ) X-Spam-Score: -0.4 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1041 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.2 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This provides the ability for architectures to enable kernel stack base address offset randomization. This feature is controlled by the boot param "randomize_kstack_offset=on/off", with its default value set by CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. This feature is based on the original idea from the last public release of PaX's RANDKSTACK feature: https://pax.grsecurity.net/docs/randkstack.txt All the credit for the original idea goes to the PaX team. Note that the design and implementation of this upstream randomize_kstack_offset feature differs greatly from the RANDKSTACK feature (see below). Reasoning for the feature: This feature aims to make harder the various stack-based attacks that rely on deterministic stack structure. We have had many such attacks in past (just to name few): https://jon.oberheide.org/files/infiltrate12-thestackisback.pdf https://jon.oberheide.org/files/stackjacking-infiltrate11.pdf https://googleprojectzero.blogspot.com/2016/06/exploiting-recursion-in-linux-kernel_20.html As Linux kernel stack protections have been constantly improving (vmap-based stack allocation with guard pages, removal of thread_info, STACKLEAK), attackers have had to find new ways for their exploits to work. They have done so, continuing to rely on the kernel's stack determinism, in situations where VMAP_STACK and THREAD_INFO_IN_TASK_STRUCT were not relevant. For example, the following recent attacks would have been hampered if the stack offset was non-deterministic between syscalls: https://repositorio-aberto.up.pt/bitstream/10216/125357/2/374717.pdf (page 70: targeting the pt_regs copy with linear stack overflow) https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html (leaked stack address from one syscall as a target during next syscall) The main idea is that since the stack offset is randomized on each system call, it is harder for an attack to reliably land in any particular place on the thread stack, even with address exposures, as the stack base will change on the next syscall. Also, since randomization is performed after placing pt_regs, the ptrace-based approach[1] to discover the randomized offset during a long-running syscall should not be possible. Design description: During most of the kernel's execution, it runs on the "thread stack", which is pretty deterministic in its structure: it is fixed in size, and on every entry from userspace to kernel on a syscall the thread stack starts construction from an address fetched from the per-cpu cpu_current_top_of_stack variable. The first element to be pushed to the thread stack is the pt_regs struct that stores all required CPU registers and syscall parameters. Finally the specific syscall function is called, with the stack being used as the kernel executes the resulting request. The goal of randomize_kstack_offset feature is to add a random offset after the pt_regs has been pushed to the stack and before the rest of the thread stack is used during the syscall processing, and to change it every time a process issues a syscall. The source of randomness is currently architecture-defined (but x86 is using the low byte of rdtsc()). Future improvements for different entropy sources is possible, but out of scope for this patch. As suggested by Andy Lutomirski, the offset is added using alloca() and an empty asm() statement with an output constraint, since it avoid changes to assembly syscall entry code, to the unwinder, and provides correct stack alignment as defined by the compiler. In order to make this available by default with zero performance impact for those that don't want it, it is boot-time selectable with static branches. This way, if the overhead is not wanted, it can just be left turned off with no performance impact. The generated assembly for x86_64 with GCC looks like this: ... ffffffff81003977: 65 8b 05 02 ea 00 7f mov %gs:0x7f00ea02(%rip),%eax # 12380 ffffffff8100397e: 25 ff 03 00 00 and $0x3ff,%eax ffffffff81003983: 48 83 c0 0f add $0xf,%rax ffffffff81003987: 25 f8 07 00 00 and $0x7f8,%eax ffffffff8100398c: 48 29 c4 sub %rax,%rsp ffffffff8100398f: 48 8d 44 24 0f lea 0xf(%rsp),%rax ffffffff81003994: 48 83 e0 f0 and $0xfffffffffffffff0,%rax ... As a result of the above stack alignment, this patch introduces about 5 bits of randomness after pt_regs is spilled to the thread stack on x86_64, and 6 bits on x86_32 (since its has 1 fewer bit required for stack alignment). The amount of entropy could be adjusted based on how much of the stack space we wish to trade for security. My measure of syscall performance overhead (on x86_64): lmbench: /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_syscall -N 10000 null randomize_kstack_offset=y Simple syscall: 0.7082 microseconds randomize_kstack_offset=n Simple syscall: 0.7016 microseconds So, roughly 0.9% overhead growth for a no-op syscall, which is very manageable. And for people that don't want this, it's off by default. There are two gotchas with using the alloca() trick. First, compilers that have Stack Clash protection (-fstack-clash-protection) enabled by default (e.g. Ubuntu[3]) add pagesize stack probes to any dynamic stack allocations. While the randomization offset is always less than a page, the resulting assembly would still contain (unreachable!) probing routines, bloating the resulting assembly. To avoid this, -fno-stack-clash-protection is unconditionally added to the kernel Makefile since this is the only dynamic stack allocation in the kernel (now that VLAs have been removed) and it is provably safe from Stack Clash style attacks. The second gotcha with alloca() is a negative interaction with -fstack-protector-strong, in that it see the alloca() as an array allocation, which triggers the unconditional addition of the stack canary function pre/post-amble which slows down syscalls regardless of the static branch. In order to avoid adding this unneeded check and its associated performance impact, architectures need to downgrade uses of -fstack-protector-strong to -fstack-protector (which only triggers for char arrays) in the compilation units that use the add_random_kstack() macro and to audit the resulting stack mitigation coverage (to make sure no desired coverage disappears). This is done in the next patches for x86 and arm64. There is, unfortunately, no attribute that can be used to disable stack protector for specific functions. Comparison to PaX RANDKSTACK feature: The RANDKSTACK feature randomizes the location of the stack start (cpu_current_top_of_stack), i.e. including the location of pt_regs structure itself on the stack. Initially this patch followed the same approach, but during the recent discussions[2], it has been determined to be of a little value since, if ptrace functionality is available for an attacker, they can use PTRACE_PEEKUSR/PTRACE_POKEUSR to read/write different offsets in the pt_regs struct, observe the cache behavior of the pt_regs accesses, and figure out the random stack offset. Another difference is that the random offset is stored in a per-cpu variable, rather than having it be per-thread. As a result, these implementations differ a fair bit in their implementation details and results, though obviously the intent is similar. [1] https://lore.kernel.org/kernel-hardening/2236FBA76BA1254E88B949DDB74E612BA4BC57C1@IRSMSX102.ger.corp.intel.com/ [2] https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ [3] https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html Co-developed-by: Elena Reshetova Signed-off-by: Elena Reshetova Link: https://lore.kernel.org/r/20190415060918.3766-1-elena.reshetova@intel.com Signed-off-by: Kees Cook --- Makefile | 4 ++++ arch/Kconfig | 23 ++++++++++++++++++ include/linux/randomize_kstack.h | 40 ++++++++++++++++++++++++++++++++ init/main.c | 23 ++++++++++++++++++ 4 files changed, 90 insertions(+) create mode 100644 include/linux/randomize_kstack.h diff --git a/Makefile b/Makefile index 4d0711f54047..1d4a8b9a6b02 100644 --- a/Makefile +++ b/Makefile @@ -779,6 +779,10 @@ ifdef CONFIG_INIT_STACK_ALL KBUILD_CFLAGS += -ftrivial-auto-var-init=pattern endif +# While VLAs have been removed, GCC produces unreachable stack probes +# for the randomize_kstack_offset feature. Disable it for all compilers. +KBUILD_CFLAGS += $(call cc-option,-fno-stack-clash-protection,) + DEBUG_CFLAGS := $(call cc-option, -fno-var-tracking-assignments) ifdef CONFIG_DEBUG_INFO diff --git a/arch/Kconfig b/arch/Kconfig index 17fe351cdde0..701c7d842714 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -854,6 +854,29 @@ config VMAP_STACK virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + def_bool n + help + An arch should select this symbol if it can support kernel stack + offset randomization with calls to add_random_kstack_offset() + during syscall entry and choose_random_kstack_offset() during + syscall exit. Downgrading of -fstack-protector-strong to + -fstack-protector should also be applied to the entry code and + closely examined, as the artificial stack bump looks like an array + to the compiler, so it will attempt to add canary checks regardless + of the static branch state. + +config RANDOMIZE_KSTACK_OFFSET_DEFAULT + bool "Randomize kernel stack offset on syscall entry" + depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + help + The kernel stack offset can be randomized (after pt_regs) by + roughly 5 bits of entropy, frustrating memory corruption + attacks that depend on stack address determinism or + cross-syscall address exposures. This feature is controlled + by kernel boot param "randomize_kstack_offset=on/off", and this + config chooses the default boot state. + config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kstack.h new file mode 100644 index 000000000000..1df0dc52cadc --- /dev/null +++ b/include/linux/randomize_kstack.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_RANDOMIZE_KSTACK_H +#define _LINUX_RANDOMIZE_KSTACK_H + +#include +#include +#include + +DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DECLARE_PER_CPU(u32, kstack_offset); + +/* + * Do not use this anywhere else in the kernel. This is used here because + * it provides an arch-agnostic way to grow the stack with correct + * alignment. Also, since this use is being explicitly masked to a max of + * 10 bits, stack-clash style attacks are unlikely. For more details see + * "VLAs" in Documentation/process/deprecated.rst + */ +void *__builtin_alloca(size_t size); + +#define add_random_kstack_offset() do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + u8 *ptr = __builtin_alloca(offset & 0x3FF); \ + asm volatile("" : "=m"(*ptr)); \ + } \ +} while (0) + +#define choose_random_kstack_offset(rand) do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + offset ^= (rand); \ + this_cpu_write(kstack_offset, offset); \ + } \ +} while (0) + +#endif diff --git a/init/main.c b/init/main.c index ee4947af823f..78fe3aea00b0 100644 --- a/init/main.c +++ b/init/main.c @@ -777,6 +777,29 @@ static void __init mm_init(void) pti_init(); } +#ifdef CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET +DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DEFINE_PER_CPU(u32, kstack_offset); + +static int __init early_randomize_kstack_offset(char *buf) +{ + int ret; + bool bool_result; + + ret = kstrtobool(buf, &bool_result); + if (ret) + return ret; + + if (bool_result) + static_branch_enable(&randomize_kstack_offset); + else + static_branch_disable(&randomize_kstack_offset); + return 0; +} +early_param("randomize_kstack_offset", early_randomize_kstack_offset); +#endif + void __init __weak arch_call_rest_init(void) { rest_init(); From patchwork Mon Apr 6 23:16:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11476845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 755571744 for ; Mon, 6 Apr 2020 23:16:58 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3141720753 for ; Mon, 6 Apr 2020 23:16:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="b71g+jph"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="FAUou8gY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3141720753 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1ofmktIJRyEvGZ4qqNRrkzJl3Cp3//LrQyKeUv7bq94=; b=b71g+jphASZr4E 2k63UoNlnGtq53frF3sl0MW3pSEaQXPHttfpUHY93ZyBRcOnU/lCqOUDsFzx5/DjTTcApTIfQiEP0 fHakRB4++4x5+F5ErasvTg/eh/dfY46EWicINnHE1SawrillKGFkQWuBg4AvNhbsQfgeI7EoSYppq 4+xHk4wqKJ3SPoqLoULKgm32d+ce1YiN0HoYq1/ZF9utfR5b6Zc4uJscpvhR6GJXQ7JHDBoiEOK5t k+wY/fBvaCy4gwL705TYbV1O59264KVPJqLnGt7gawOtvH7X+n+da/gRvgeoSzESU7SAMdAXeToKj 273yfviwanL6uGOBV01g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLaz4-0004Hb-NP; Mon, 06 Apr 2020 23:16:50 +0000 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayU-0003ds-K8 for linux-arm-kernel@lists.infradead.org; Mon, 06 Apr 2020 23:16:16 +0000 Received: by mail-pl1-x642.google.com with SMTP id e1so479465plt.9 for ; Mon, 06 Apr 2020 16:16:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=afO1zKmkq1sj5aqNSAPmo8IRjezuEuKjnM3GlRCWtRE=; b=FAUou8gYmt2MCxO2cKyXhzSw4j1IC8KgeC+bXiqTqe5QqSNRmdnqOuVJoHQDJ+UyPU RdIafVDCCHUGnepHULs0h0QZtUehhYm52xPc8JIfEafNjj7L97tUHTMzVrIHdX23tu7T 870tT8w8N8DbhdKYHGf474/jG4q9lS98mHNd0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=afO1zKmkq1sj5aqNSAPmo8IRjezuEuKjnM3GlRCWtRE=; b=fursPZ7e+5FM8E25YFRnTICzcGvmw5oVkjpCG5LlH8pgjrg9kFmn7QK2KHWkrdaZaE ouRbjajoZQHlTyAmRkEJ37ccQypWKe3f4aNshDpQLMtvnmODuvhcfzCi1Z1GR8qbQ+hQ 4gu/8Hf1BldFyq0OCJTnEPC3z0V35fxIto0u6bnWeFQ5SPCfYWt4WzFsk81Pb5y0RHEE g8ici7IdvUzK2fpeAy4Ffqwlq8LHyLLXR3G7AXXe24AUvZkYTUiBOQ8FFfW2Y790EgaX gHpR+6HxPRssigpcnCk0VD4VcMWf+jalFS4gQhA/UZKui8RSEIvEwnq/7+hAE6gWm+xV lwQg== X-Gm-Message-State: AGi0Pua1Dr5Fh13tmrjQh8I3e2139P/uurykhMgyrsfHWZKJsMNtn6fo J1V++ePbVLTF8br1aEC5KE3S2A== X-Google-Smtp-Source: APiQypLKqmE7eCIX43I8YR8JbiR94l3ZUUPoGei30hzmH3c+zDO2qS2G3j+Tlv/Caq6mWLyP6sMzhQ== X-Received: by 2002:a17:902:788e:: with SMTP id q14mr22301315pll.72.1586214974066; Mon, 06 Apr 2020 16:16:14 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id f8sm10635773pgc.75.2020.04.06.16.16.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Apr 2020 16:16:10 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v3 4/5] x86/entry: Enable random_kstack_offset support Date: Mon, 6 Apr 2020 16:16:05 -0700 Message-Id: <20200406231606.37619-5-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200406231606.37619-1-keescook@chromium.org> References: <20200406231606.37619-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200406_161614_667912_62D1C863 X-CRM114-Status: GOOD ( 15.22 ) X-Spam-Score: -0.4 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:642 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.2 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. In order to avoid unconditional stack canaries on syscall entry, also downgrade from -fstack-protector-strong to -fstack-protector to avoid triggering checks due to alloca(). Examining the resulting canary coverage changes to common.o, this also removes canaries in other functions, due to a handful of declarations of "__u64 args[6]" (from seccomp) and "unsigned long args[6]" (from tracepoints), but their accesses are indexed (instead of via dynamically sized linear reads/writes) so the risk of removing useful mitigation coverage here is very low. Signed-off-by: Kees Cook --- arch/x86/Kconfig | 1 + arch/x86/entry/Makefile | 9 +++++++++ arch/x86/entry/common.c | 12 +++++++++++- 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index beea77046f9b..b9d449581eb6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -150,6 +150,7 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE diff --git a/arch/x86/entry/Makefile b/arch/x86/entry/Makefile index 06fc70cf5433..7b40e6ae2618 100644 --- a/arch/x86/entry/Makefile +++ b/arch/x86/entry/Makefile @@ -7,6 +7,15 @@ OBJECT_FILES_NON_STANDARD_entry_64_compat.o := y CFLAGS_syscall_64.o += $(call cc-option,-Wno-override-init,) CFLAGS_syscall_32.o += $(call cc-option,-Wno-override-init,) + +# Downgrade to -fstack-protector to avoid triggering unneeded stack canary +# checks due to randomize_kstack_offset. This also removes canaries in +# other places as well, due to a handful of declarations of __u64 args[6] +# (seccomp) and unsigned long args[6] (tracepoints), but their accesses +# are indexed (instead of via dynamically sized linear reads/writes) so +# the risk of removing useful mitigation coverage here is very low. +CFLAGS_common.o += $(subst -fstack-protector-strong,-fstack-protector,$(filter -fstack-protector-strong,$(KBUILD_CFLAGS))) + obj-y := entry_$(BITS).o thunk_$(BITS).o syscall_$(BITS).o obj-y += common.o diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 9747876980b5..086d7af570af 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -189,6 +190,13 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs) lockdep_assert_irqs_disabled(); lockdep_sys_exit(); + /* + * x86_64 stack alignment means 3 bits are ignored, so keep + * the top 5 bits. x86_32 needs only 2 bits of alignment, so + * the top 6 bits will be used. + */ + choose_random_kstack_offset(rdtsc() & 0xFF); + cached_flags = READ_ONCE(ti->flags); if (unlikely(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) @@ -283,6 +291,7 @@ __visible void do_syscall_64(unsigned long nr, struct pt_regs *regs) { struct thread_info *ti; + add_random_kstack_offset(); enter_from_user_mode(); local_irq_enable(); ti = current_thread_info(); @@ -355,6 +364,7 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs) /* Handles int $0x80 */ __visible void do_int80_syscall_32(struct pt_regs *regs) { + add_random_kstack_offset(); enter_from_user_mode(); local_irq_enable(); do_syscall_32_irqs_on(regs); @@ -378,8 +388,8 @@ __visible long do_fast_syscall_32(struct pt_regs *regs) */ regs->ip = landing_pad; + add_random_kstack_offset(); enter_from_user_mode(); - local_irq_enable(); /* Fetch EBP from where the vDSO stashed it. */ From patchwork Mon Apr 6 23:16:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11476853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E55D714DD for ; Mon, 6 Apr 2020 23:17:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A4473206C0 for ; Mon, 6 Apr 2020 23:17:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="F1bANOUH"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="SHvDMZKy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4473206C0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GcxxVbYRHqyiofyelJE+49dnE2lsAbXwNxhJgKCegTg=; b=F1bANOUHU4ySJh JaSaZKhDKe7agzfmtmmxqyG12e3AjpV9jjGo396P8Mckkh9Vt/5gwo4/sAnReb+ULO/pnvDSOyDHa /vbW9YvHtDLIRx/PQA12FJ4Yg8Gw+27DU/0wHLwkmHhvukeWJxiPDW8KxEnH3Tw3xtguT/dpFJM/H pV9xc+aAbPLrWGoWSc69gNDnL0dvHd29VYErhz5u/Mjh+KQoFw4ci2HY7bqKYOEs6HjGF4TcvKeBF xnWsx2+5ZVOTQTdg4JFfLs0VLZt52ic+BS91rm79WbvWuu/g7lY7SENHn3AuOHEX6OliPrc+VNMo9 NveAXhP6bo9hJ4OtT6Eg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLazc-0004nR-3q; Mon, 06 Apr 2020 23:17:24 +0000 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jLayW-0003eV-Bz for linux-arm-kernel@lists.infradead.org; Mon, 06 Apr 2020 23:16:17 +0000 Received: by mail-pf1-x443.google.com with SMTP id q3so8346923pff.13 for ; Mon, 06 Apr 2020 16:16:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6T/7mkz2QJj1i6yBI9rf5y5UrGkfIoNdamdak5bIEr8=; b=SHvDMZKynQoupRAYdyzGP7833YCcGwM4Pjmu4hIlfgEi41IUoIMZIvMFkCimFNo1pC ukiFfjPKn6dwME2ko+g8mNG5D99lrZbhcV9idRFBRE7QuhQxO7LJZsbFMVIn38ydgfCG 4qMfQRVb21FqPszfB/2JLkg9/aNX0WWnHNT0E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6T/7mkz2QJj1i6yBI9rf5y5UrGkfIoNdamdak5bIEr8=; b=jKt0MP7xO5kLY040SQsqUB86dNlpSeJbiSrg8qrOFMDtu/3xg5aqMIUjIDcuLefqh7 f82nWPldm0+AomaB24U/v4XA08Em60jyJb/1upOE087MMnuREyXihomeWsNOx859xNNY qPgT8xPEBRyfu1/cP+3tZAQwJNIOHTb26n5NF6WYdAIphrLUjn6ADkJJY5A7s4PAGCOg KaG6qdSxrcYMAnD4VePVplCramzFWlejyZGia1yM2EsNM1pXJENfoRxfjqlIRS7bX8iM xlit+FWBWs9TIYsRQ9lXuouupEVFSZLbOt9XKpyrQIoMmS/z7tqixa8X0fDsx9LOO35Q 8fMQ== X-Gm-Message-State: AGi0PuYdYvXBh6zA3xBGvVfrIZ3BHJppxb5CPzzMEXR16CHeXGv094iK oAHedf87yerTfewo554pQ4tLXg== X-Google-Smtp-Source: APiQypKPb2SK+P7H+McMjGGqVwR7DToHZCme/uHbB59Pjk0ShbX/p0nUYaBfdUBZDv7/4hnnvCIAMQ== X-Received: by 2002:a62:d10b:: with SMTP id z11mr1732920pfg.205.1586214975509; Mon, 06 Apr 2020 16:16:15 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id k20sm11602048pgn.62.2020.04.06.16.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Apr 2020 16:16:14 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v3 5/5] arm64: entry: Enable random_kstack_offset support Date: Mon, 6 Apr 2020 16:16:06 -0700 Message-Id: <20200406231606.37619-6-keescook@chromium.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200406231606.37619-1-keescook@chromium.org> References: <20200406231606.37619-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200406_161616_451335_8D292CC5 X-CRM114-Status: GOOD ( 14.98 ) X-Spam-Score: -0.4 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:443 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.2 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. In order to avoid unconditional stack canaries on syscall entry, also downgrade from -fstack-protector-strong to -fstack-protector to avoid triggering checks due to alloca(). Examining the resulting syscall.o, sees no changes in canary coverage (none before, none now). Signed-off-by: Kees Cook --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/Makefile | 4 ++++ arch/arm64/kernel/syscall.c | 10 ++++++++++ 3 files changed, 15 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 0b30e884e088..4d5aa4959f72 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -127,6 +127,7 @@ config ARM64 select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT select HAVE_ARCH_PREL32_RELOCATIONS + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_STACKLEAK select HAVE_ARCH_THREAD_STRUCT_WHITELIST diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index fc6488660f64..b89005f125d6 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -11,6 +11,10 @@ CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) +# Downgrade to -fstack-protector to avoid triggering unneeded stack canary +# checks due to randomize_kstack_offset. +CFLAGS_syscall.o += $(subst -fstack-protector-strong,-fstack-protector,$(filter -fstack-protector-strong,$(KBUILD_CFLAGS))) + # Object file lists. obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ entry-common.o entry-fpsimd.o process.o ptrace.o \ diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index a12c0c88d345..238dbd753b44 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,8 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, { long ret; + add_random_kstack_offset(); + if (scno < sc_nr) { syscall_fn_t syscall_fn; syscall_fn = syscall_table[array_index_nospec(scno, sc_nr)]; @@ -51,6 +54,13 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, } regs->regs[0] = ret; + + /* + * Since the compiler chooses a 4 bit alignment for the stack, + * let's save one additional bit (9 total), which gets us up + * near 5 bits of entropy. + */ + choose_random_kstack_offset(get_random_int() & 0x1FF); } static inline bool has_syscall_work(unsigned long flags)