From patchwork Mon Jun 22 19:31:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11618849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E9A714B7 for ; Mon, 22 Jun 2020 19:33:49 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2860020776 for ; Mon, 22 Jun 2020 19:33:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="wiS15wFa"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="i8xeTUYk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2860020776 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2jnD/DuRxSmyz2hSXHtVQD86qrY9YGcW6jx+i7NKms0=; b=wiS15wFa5tMJ2Re3DhY2dRuDm 0+PVZx4+wEDGGiBPl8jb4zEFIsgXG9FLxkElzpW59ap1wPzTTpf+5Wyl2yb+FX1HDnwON3f4PsBJL iQiZLVOQzJW8bUf5CDmOMeFsjNQeAt1gvwvGY4IPh04qdZ3R4ssuZKWwde4bLB4M07kImPVT8jAto ZowAkZ+oY3i/owzyXm6kNzDcxATewWGWYx2vTRU/Xts/vR00RcAaghY6BAxkUrcE9t0uPQD7JuBTG FgeSwV7EEjy3yGr8O4gqxyCFUgPolyZ7j2FuRm513hS/6xUPQJGxRCMiKi2BqGajMXemagOpPRD9z DiwfGvBzg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAh-00021H-0R; Mon, 22 Jun 2020 19:31:59 +0000 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAc-0001zj-9E for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 19:31:55 +0000 Received: by mail-pl1-x641.google.com with SMTP id s14so3911582plq.6 for ; Mon, 22 Jun 2020 12:31:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4nbp+NPCtqAP+1aCZB+BxvvOvC8Ydbr5MOs46Iqp71U=; b=i8xeTUYkhUyZD/ZejYoQPEhDfLdlYyCVxvPsD8xQpm+zO42lEdDiJvk8BDYgGEcuJQ qLArX4e0fAxdqKPkGHLw5VUwjVETfJ760S8EqnIUxpKn48TOqu7uedsiGT2X2OlwBuzf rL0LTOtoD7Ifjax5kOpx8Vz+vl26lvwToxEYs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4nbp+NPCtqAP+1aCZB+BxvvOvC8Ydbr5MOs46Iqp71U=; b=qfIWcSJ7Lsms1YHaq2gZva9rCiWuQtIyIWsUr/uwRmhUecJTBYmAbX98kUhPMbevtv 9ok4T480N/WF6A/WMDElmXbM9KuN7SHqj+fuL3CCP/3HF9uUpwyuvNQae7gK8MxjIIss WEdpDuvagF8Pocc0R0LTPb5jH8CoE/zeCllkI2+SAyoImqkItnA+5jQUSE4H8jCSatdh giZo/ZcnBXP34G8tEkXXi5fLTxDkbgdKuiEZgwp836VTdYYjnRHUloAAeTWeJVeUzimA as5hsXubZHLCwda4ikhGG2OvwDWJX/zG4mFkGI0yxnA9I0ypTPe2d9hhRLNWe51Idxuj Tx0w== X-Gm-Message-State: AOAM532yn5O33x7QnETysSR2PNAe6GvR6kOMLFOEHQtZ4IYWBq1QaAWp BpigicaLd1NQ0s3lzaLgyH/b7g== X-Google-Smtp-Source: ABdhPJx7b/cjaG3iRKt8wz6ljAUXxwMmy0SXSiUuqqLbeMskXHHKqA0MuLk0OKzDDvDp/IX6avXBCw== X-Received: by 2002:a17:90a:5a07:: with SMTP id b7mr18992351pjd.130.1592854311856; Mon, 22 Jun 2020 12:31:51 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id q22sm14396934pfg.192.2020.06.22.12.31.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:31:50 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v4 1/5] jump_label: Provide CONFIG-driven build state defaults Date: Mon, 22 Jun 2020 12:31:42 -0700 Message-Id: <20200622193146.2985288-2-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200622193146.2985288-1-keescook@chromium.org> References: <20200622193146.2985288-1-keescook@chromium.org> MIME-Version: 1.0 X-Spam-Note: CRM114 invocation failed X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:641 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova , Alexander Popov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Choosing the initial state of static branches changes the assembly layout (if the condition is expected to be likely, inline, or unlikely, out of line via a jump). A few places in the kernel use (or could be using) a CONFIG to choose the default state, so provide the infrastructure to do this and convert the existing cases (init_on_alloc and init_on_free) to the new macros. Acked-by: Peter Zijlstra (Intel) Signed-off-by: Kees Cook --- include/linux/jump_label.h | 19 +++++++++++++++++++ include/linux/mm.h | 12 ++---------- mm/page_alloc.c | 12 ++---------- 3 files changed, 23 insertions(+), 20 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 3526c0aee954..615fdfb871a3 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -382,6 +382,21 @@ struct static_key_false { [0 ... (count) - 1] = STATIC_KEY_FALSE_INIT, \ } +#define _DEFINE_STATIC_KEY_1(name) DEFINE_STATIC_KEY_TRUE(name) +#define _DEFINE_STATIC_KEY_0(name) DEFINE_STATIC_KEY_FALSE(name) +#define DEFINE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_, IS_ENABLED(cfg))(name) + +#define _DEFINE_STATIC_KEY_RO_1(name) DEFINE_STATIC_KEY_TRUE_RO(name) +#define _DEFINE_STATIC_KEY_RO_0(name) DEFINE_STATIC_KEY_FALSE_RO(name) +#define DEFINE_STATIC_KEY_MAYBE_RO(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_RO_, IS_ENABLED(cfg))(name) + +#define _DECLARE_STATIC_KEY_1(name) DECLARE_STATIC_KEY_TRUE(name) +#define _DECLARE_STATIC_KEY_0(name) DECLARE_STATIC_KEY_FALSE(name) +#define DECLARE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DECLARE_STATIC_KEY_, IS_ENABLED(cfg))(name) + extern bool ____wrong_branch_error(void); #define static_key_enabled(x) \ @@ -482,6 +497,10 @@ extern bool ____wrong_branch_error(void); #endif /* CONFIG_JUMP_LABEL */ +#define static_branch_maybe(config, x) \ + (IS_ENABLED(config) ? static_branch_likely(x) \ + : static_branch_unlikely(x)) + /* * Advanced usage; refcount, branch is enabled when: count != 0 */ diff --git a/include/linux/mm.h b/include/linux/mm.h index dc7b87310c10..0e6824fd4458 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2889,11 +2889,7 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_alloc); -#else -DECLARE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc) && @@ -2902,11 +2898,7 @@ static inline bool want_init_on_alloc(gfp_t flags) return flags & __GFP_ZERO; } -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_free); -#else -DECLARE_STATIC_KEY_FALSE(init_on_free); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { return static_branch_unlikely(&init_on_free) && diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 48eb0f1410d4..5885a612fa18 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -136,18 +136,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_alloc); -#else -DEFINE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_free); -#else -DEFINE_STATIC_KEY_FALSE(init_on_free); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static int __init early_init_on_alloc(char *buf) From patchwork Mon Jun 22 19:31:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11618847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12750912 for ; Mon, 22 Jun 2020 19:33:49 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DFCB820776 for ; Mon, 22 Jun 2020 19:33:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="EQZimtnA"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="jQvwT91a" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DFCB820776 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WiJIsZ3FhXbSZuX509uP/1onvvdjOmaPGHZ+gJ9GJT0=; b=EQZimtnA72AJ72NWpIi0e4p8c bLreeeRBI1bPJLL6C5zVUKRzHxDpvWRZkEAHRP1roBWa5cK7GQQfjMjDG6lIgK99cSh8FHH9aw7lu r/9OjCR1GFBqxBEH3He/5kTCID6b17WPt2KJ1RqUoxoDbwwzMHj6dtlt7g+Bma/hsfGXOmsY5iG/c G9spYAKGeLm2YOJdcOxVJ+rizeCYFcemR2IN2xcFtu68yIbR4Xa2wcrItNbtvMs6aPn6Z6rWDz4pp juXYzCW3HFgwlOaaRQIolxF35j3374un1RtUVW3RfDl/J8+j1eGxgOHYVHqAZF7YVVyArmc7Z5lrX ePzd3SVCg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAl-00022G-0F; Mon, 22 Jun 2020 19:32:03 +0000 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAc-0001zg-Ka for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 19:31:57 +0000 Received: by mail-pf1-x443.google.com with SMTP id x22so8865975pfn.3 for ; Mon, 22 Jun 2020 12:31:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bd9f3BBqo3zGwwTfmvjboaNS4RxuJ7DCwqiU44HfZsQ=; b=jQvwT91aNdu6DbbTFmcHtowv1QDuWeDxsDk/RqKsPvwbGbHTpYzrDrmmY/0dq+aBJQ jlwYwDhLmthT175NDbEj6C17KQJgc9BEyT+EwcD3Md/NbUgFJ+PR+YCJt9RkY1QSZdRZ QY9d67wu3uA3m5bOy/gHJd8An71KIaVS3oaeE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bd9f3BBqo3zGwwTfmvjboaNS4RxuJ7DCwqiU44HfZsQ=; b=Y0ZmL4Z7c8oAhkeK0oMFDkjJl+AWOQxv+ntMxRacODL+TzePEdUvrMBTlA8WHPayeV FkGbM89mxzbEzkcT/2nGZNI23ofhoWv13Faz2wxAC5ci0NTfw7N+YfUaoz923tzeODOk yb8mvCjpUd0DX5oLT3p28A0goksKeBJAOPDT+LdEpBVKnTN7WZ5aprru4w6lrAjk3jjq hGNttbqanlcGNOAm+xE1jxLbecj+O2YNbGTEhL2BJJWIoBKcA/cbpDw28U+GcDA6WvF5 ++IguLJ1ckveVC/2Xn/ah+u0KiZjoiJkwb/zq9Uz6lj4ZJGgN8pMYpas+V+IzNxYjnGI H37Q== X-Gm-Message-State: AOAM531K5opgiryYlIhsVx944PnGPa+NxspcBEZt6ImxCSp9SKDxsXO8 9w4bqP75PECO5DYMQVJ9H59NOQ== X-Google-Smtp-Source: ABdhPJy/XsCwbSTQ8GDNdy6eIPIN39i7/vLM9SHQLKSs+iIR061Tg/mVW3fZLxYeE/B4wCdXiH9v5A== X-Received: by 2002:a62:1d0b:: with SMTP id d11mr15185345pfd.1.1592854312494; Mon, 22 Jun 2020 12:31:52 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id y18sm14064284pfn.177.2020.06.22.12.31.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:31:50 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v4 2/5] init_on_alloc: Unpessimize default-on builds Date: Mon, 22 Jun 2020 12:31:43 -0700 Message-Id: <20200622193146.2985288-3-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200622193146.2985288-1-keescook@chromium.org> References: <20200622193146.2985288-1-keescook@chromium.org> MIME-Version: 1.0 X-Spam-Note: CRM114 invocation failed X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:443 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova , Alexander Popov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Right now, the state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branch tests. Use the new jump_label macro to check CONFIG settings to default to the "expected" state, unpessimizes the resulting assembly code. Reviewed-by: Alexander Potapenko Signed-off-by: Kees Cook --- include/linux/mm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0e6824fd4458..0a05b20870c2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2892,7 +2892,8 @@ static inline void kernel_poison_pages(struct page *page, int numpages, DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc) && + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc) && !page_poisoning_enabled()) return true; return flags & __GFP_ZERO; @@ -2901,7 +2902,8 @@ static inline bool want_init_on_alloc(gfp_t flags) DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free) && + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free) && !page_poisoning_enabled(); } From patchwork Mon Jun 22 19:31:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11618855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2B4114B7 for ; Mon, 22 Jun 2020 19:34:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BE89C207DD for ; Mon, 22 Jun 2020 19:34:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="jrE2a2dQ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="qYyAvp99"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="ajsr+H7m" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE89C207DD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WRHof7PSGAGr7iL5Kl6t70vu+BmjQTb3kA4HYyujGWE=; b=jrE2a2dQJH13Sgk4OzPjVyic/ yO+f4MWC70i2Yu4vG4Sig/DLZGO0gRlsQVZSmjr5TwLBEfCs8LzbcQc+p/NegfAIA5tO+IqzI1tO5 24EoEEJxXcXE2pGUsNWY6Nhh+9Xy5BMtZPgBATjzQN8eZUHqPC6AzU/EDq1qrBks3s6xuYZxXvVFs IFns1tgCZfOBL9b+lWHw8qVaNSbp5AmgTkP6rCqYPmF94mRFug0Cn29S6T8JMqnp6GX0yCcHS5XgJ VRD026bsjdRkzsMGFRj7NrPwmw/AOnV4hHRmucDiOclW2FwyzSI9EYYqSQJ87vvKe75IdfbFZpPsq Ksydxlv2g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAs-00024c-11; Mon, 22 Jun 2020 19:32:10 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAk-00022K-Tt for linux-arm-kernel@merlin.infradead.org; Mon, 22 Jun 2020 19:32:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ETibkmg3LTh2K86BfakY+sfkjiZxxEduxUu4TP2wXYU=; b=qYyAvp99cboL2LP8hUh2y0cDl+ p+xvlNNAcQIqwj65oNr0cz+vGuIRLtmQakeBkoKmnCAKulI/WFxDa9JxD7F5Z/Bvai2VgTXoVJAb6 YfhHAEaoEVW4kvccYTYRMGpt5BjtxixpsyU/RQj5Sp7zetIuMXrDR5NWXVMV4oPwYUoMljyXT4fcq SYodAIei8nxgNIRrgOAE5TM9B071vgmw65oBp72qw1yoZq6AS9IY1Pig48veHmnuOOJuwKj8J3mAW H0pkQ9KeYNQtNL9bJqPCOlgyteAzmMkSpJ9Hks6abWu5K1Rdl/1bwjEaAZBmnDd2bA3Jxpuu2PhuE WDrJIusQ==; Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAf-0002TZ-CO for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 19:32:02 +0000 Received: by mail-pj1-x1041.google.com with SMTP id cm23so291563pjb.5 for ; Mon, 22 Jun 2020 12:31:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ETibkmg3LTh2K86BfakY+sfkjiZxxEduxUu4TP2wXYU=; b=ajsr+H7mMTpamQsCtpYBipO36zKVs/1VLjBK7TBKRWLKNdVO8ZKynduIYcvzWHc9js Z35TrOXmUYoN58DRLVwW/W5I2jmNcalB8MrJEag4JPhug2KIS8ZCMZFlbSnTnngiJLG0 SS6ZVaN1lxd1tUDPgKdG5ys2IDPGx9XGZrzn4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ETibkmg3LTh2K86BfakY+sfkjiZxxEduxUu4TP2wXYU=; b=eUn3VXK4vyo8ynFoXDNaARNuKPv0Tj5KE0+WR7SnVip+J3fCMxHURa7rezCsv/Xuq3 nDttU8I3eAE25VBGRTvlf1xKwg+LpBS5gYEZwoPhl7EOrGn37QwoSRHG4fHQoX7xVn7Q FgaavN38j18wdjUDcX+09/QiHThZbiaHj16kzI3TllPBpXcHuABbVQq4/CZ3/JvIWO9a OtAgVdDNfTZYRoayPflSll4OzyQo0vA11E0CMPbqykvi14PePg+1JrBdI88SC/ytHN9H AUi5aT2IOzB8jqzdfWk+ImbJKMz2ehhtrzUldDQ44WdJjq02rpLhVmkhJhepyJeUUipF a9bg== X-Gm-Message-State: AOAM532xxnN2npgr3wOaRMRZTZslK6rzH6KlJw5/YXQDZpmZaQIdBezS NKp4TbAuIC7klLnDZEdxWVccWIkoalw= X-Google-Smtp-Source: ABdhPJy6fJbiU1P7tQzm3n6EQcYtlh6IRV4Fjc3C1YFRCQw+vvJVEo6c02AaYhufYUaHTMrSi0vv4A== X-Received: by 2002:a17:90a:fa95:: with SMTP id cu21mr19155151pjb.56.1592854313991; Mon, 22 Jun 2020 12:31:53 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id e128sm14613580pfe.196.2020.06.22.12.31.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:31:51 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v4 3/5] stack: Optionally randomize kernel stack offset each syscall Date: Mon, 22 Jun 2020 12:31:44 -0700 Message-Id: <20200622193146.2985288-4-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200622193146.2985288-1-keescook@chromium.org> References: <20200622193146.2985288-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200622_203157_912655_A2DCFDD7 X-CRM114-Status: GOOD ( 43.73 ) X-Spam-Score: -2.1 (--) X-Spam-Report: SpamAssassin version 3.4.4 on casper.infradead.org summary: Content analysis details: (-2.1 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1041 listed in] [list.dnswl.org] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova , Alexander Popov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This provides the ability for architectures to enable kernel stack base address offset randomization. This feature is controlled by the boot param "randomize_kstack_offset=on/off", with its default value set by CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. This feature is based on the original idea from the last public release of PaX's RANDKSTACK feature: https://pax.grsecurity.net/docs/randkstack.txt All the credit for the original idea goes to the PaX team. Note that the design and implementation of this upstream randomize_kstack_offset feature differs greatly from the RANDKSTACK feature (see below). Reasoning for the feature: This feature aims to make harder the various stack-based attacks that rely on deterministic stack structure. We have had many such attacks in past (just to name few): https://jon.oberheide.org/files/infiltrate12-thestackisback.pdf https://jon.oberheide.org/files/stackjacking-infiltrate11.pdf https://googleprojectzero.blogspot.com/2016/06/exploiting-recursion-in-linux-kernel_20.html As Linux kernel stack protections have been constantly improving (vmap-based stack allocation with guard pages, removal of thread_info, STACKLEAK), attackers have had to find new ways for their exploits to work. They have done so, continuing to rely on the kernel's stack determinism, in situations where VMAP_STACK and THREAD_INFO_IN_TASK_STRUCT were not relevant. For example, the following recent attacks would have been hampered if the stack offset was non-deterministic between syscalls: https://repositorio-aberto.up.pt/bitstream/10216/125357/2/374717.pdf (page 70: targeting the pt_regs copy with linear stack overflow) https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html (leaked stack address from one syscall as a target during next syscall) The main idea is that since the stack offset is randomized on each system call, it is harder for an attack to reliably land in any particular place on the thread stack, even with address exposures, as the stack base will change on the next syscall. Also, since randomization is performed after placing pt_regs, the ptrace-based approach[1] to discover the randomized offset during a long-running syscall should not be possible. Design description: During most of the kernel's execution, it runs on the "thread stack", which is pretty deterministic in its structure: it is fixed in size, and on every entry from userspace to kernel on a syscall the thread stack starts construction from an address fetched from the per-cpu cpu_current_top_of_stack variable. The first element to be pushed to the thread stack is the pt_regs struct that stores all required CPU registers and syscall parameters. Finally the specific syscall function is called, with the stack being used as the kernel executes the resulting request. The goal of randomize_kstack_offset feature is to add a random offset after the pt_regs has been pushed to the stack and before the rest of the thread stack is used during the syscall processing, and to change it every time a process issues a syscall. The source of randomness is currently architecture-defined (but x86 is using the low byte of rdtsc()). Future improvements for different entropy sources is possible, but out of scope for this patch. As suggested by Andy Lutomirski, the offset is added using alloca() and an empty asm() statement with an output constraint, since it avoid changes to assembly syscall entry code, to the unwinder, and provides correct stack alignment as defined by the compiler. In order to make this available by default with zero performance impact for those that don't want it, it is boot-time selectable with static branches. This way, if the overhead is not wanted, it can just be left turned off with no performance impact. The generated assembly for x86_64 with GCC looks like this: ... ffffffff81003977: 65 8b 05 02 ea 00 7f mov %gs:0x7f00ea02(%rip),%eax # 12380 ffffffff8100397e: 25 ff 03 00 00 and $0x3ff,%eax ffffffff81003983: 48 83 c0 0f add $0xf,%rax ffffffff81003987: 25 f8 07 00 00 and $0x7f8,%eax ffffffff8100398c: 48 29 c4 sub %rax,%rsp ffffffff8100398f: 48 8d 44 24 0f lea 0xf(%rsp),%rax ffffffff81003994: 48 83 e0 f0 and $0xfffffffffffffff0,%rax ... As a result of the above stack alignment, this patch introduces about 5 bits of randomness after pt_regs is spilled to the thread stack on x86_64, and 6 bits on x86_32 (since its has 1 fewer bit required for stack alignment). The amount of entropy could be adjusted based on how much of the stack space we wish to trade for security. My measure of syscall performance overhead (on x86_64): lmbench: /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_syscall -N 10000 null randomize_kstack_offset=y Simple syscall: 0.7082 microseconds randomize_kstack_offset=n Simple syscall: 0.7016 microseconds So, roughly 0.9% overhead growth for a no-op syscall, which is very manageable. And for people that don't want this, it's off by default. There are two gotchas with using the alloca() trick. First, compilers that have Stack Clash protection (-fstack-clash-protection) enabled by default (e.g. Ubuntu[3]) add pagesize stack probes to any dynamic stack allocations. While the randomization offset is always less than a page, the resulting assembly would still contain (unreachable!) probing routines, bloating the resulting assembly. To avoid this, -fno-stack-clash-protection is unconditionally added to the kernel Makefile since this is the only dynamic stack allocation in the kernel (now that VLAs have been removed) and it is provably safe from Stack Clash style attacks. The second gotcha with alloca() is a negative interaction with -fstack-protector-strong, in that it sees the alloca() as an array allocation, which triggers the unconditional addition of the stack canary function pre/post-amble which slows down syscalls regardless of the static branch. In order to avoid adding this unneeded check and its associated performance impact, architectures need to downgrade uses of -fstack-protector-strong to -fstack-protector (which only triggers for char arrays) in the compilation units that use the add_random_kstack() macro and to audit the resulting stack mitigation coverage (to make sure no desired coverage disappears). This change is not needed on x86 because stack protector is already unconditionally disabled for the compilation unite, but is needed on arm64. There is, unfortunately, no attribute that can be used to disable stack protector for specific functions. Comparison to PaX RANDKSTACK feature: The RANDKSTACK feature randomizes the location of the stack start (cpu_current_top_of_stack), i.e. including the location of pt_regs structure itself on the stack. Initially this patch followed the same approach, but during the recent discussions[2], it has been determined to be of a little value since, if ptrace functionality is available for an attacker, they can use PTRACE_PEEKUSR/PTRACE_POKEUSR to read/write different offsets in the pt_regs struct, observe the cache behavior of the pt_regs accesses, and figure out the random stack offset. Another difference is that the random offset is stored in a per-cpu variable, rather than having it be per-thread. As a result, these implementations differ a fair bit in their implementation details and results, though obviously the intent is similar. [1] https://lore.kernel.org/kernel-hardening/2236FBA76BA1254E88B949DDB74E612BA4BC57C1@IRSMSX102.ger.corp.intel.com/ [2] https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ [3] https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html Co-developed-by: Elena Reshetova Signed-off-by: Elena Reshetova Link: https://lore.kernel.org/r/20190415060918.3766-1-elena.reshetova@intel.com Signed-off-by: Kees Cook --- Makefile | 4 ++++ arch/Kconfig | 23 ++++++++++++++++++ include/linux/randomize_kstack.h | 40 ++++++++++++++++++++++++++++++++ init/main.c | 23 ++++++++++++++++++ 4 files changed, 90 insertions(+) create mode 100644 include/linux/randomize_kstack.h diff --git a/Makefile b/Makefile index b46e91bf0b0e..8cb7a1388950 100644 --- a/Makefile +++ b/Makefile @@ -809,6 +809,10 @@ ifdef CONFIG_INIT_STACK_ALL KBUILD_CFLAGS += -ftrivial-auto-var-init=pattern endif +# While VLAs have been removed, GCC produces unreachable stack probes +# for the randomize_kstack_offset feature. Disable it for all compilers. +KBUILD_CFLAGS += $(call cc-option,-fno-stack-clash-protection,) + DEBUG_CFLAGS := $(call cc-option, -fno-var-tracking-assignments) ifdef CONFIG_DEBUG_INFO diff --git a/arch/Kconfig b/arch/Kconfig index 1ea61290900a..1f52c9cfefca 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -883,6 +883,29 @@ config VMAP_STACK virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + def_bool n + help + An arch should select this symbol if it can support kernel stack + offset randomization with calls to add_random_kstack_offset() + during syscall entry and choose_random_kstack_offset() during + syscall exit. Downgrading of -fstack-protector-strong to + -fstack-protector should also be applied to the entry code and + closely examined, as the artificial stack bump looks like an array + to the compiler, so it will attempt to add canary checks regardless + of the static branch state. + +config RANDOMIZE_KSTACK_OFFSET_DEFAULT + bool "Randomize kernel stack offset on syscall entry" + depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + help + The kernel stack offset can be randomized (after pt_regs) by + roughly 5 bits of entropy, frustrating memory corruption + attacks that depend on stack address determinism or + cross-syscall address exposures. This feature is controlled + by kernel boot param "randomize_kstack_offset=on/off", and this + config chooses the default boot state. + config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kstack.h new file mode 100644 index 000000000000..1df0dc52cadc --- /dev/null +++ b/include/linux/randomize_kstack.h @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_RANDOMIZE_KSTACK_H +#define _LINUX_RANDOMIZE_KSTACK_H + +#include +#include +#include + +DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DECLARE_PER_CPU(u32, kstack_offset); + +/* + * Do not use this anywhere else in the kernel. This is used here because + * it provides an arch-agnostic way to grow the stack with correct + * alignment. Also, since this use is being explicitly masked to a max of + * 10 bits, stack-clash style attacks are unlikely. For more details see + * "VLAs" in Documentation/process/deprecated.rst + */ +void *__builtin_alloca(size_t size); + +#define add_random_kstack_offset() do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + u8 *ptr = __builtin_alloca(offset & 0x3FF); \ + asm volatile("" : "=m"(*ptr)); \ + } \ +} while (0) + +#define choose_random_kstack_offset(rand) do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + offset ^= (rand); \ + this_cpu_write(kstack_offset, offset); \ + } \ +} while (0) + +#endif diff --git a/init/main.c b/init/main.c index 0ead83e86b5a..fa8ae0ae3ac2 100644 --- a/init/main.c +++ b/init/main.c @@ -822,6 +822,29 @@ static void __init mm_init(void) pti_init(); } +#ifdef CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET +DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DEFINE_PER_CPU(u32, kstack_offset); + +static int __init early_randomize_kstack_offset(char *buf) +{ + int ret; + bool bool_result; + + ret = kstrtobool(buf, &bool_result); + if (ret) + return ret; + + if (bool_result) + static_branch_enable(&randomize_kstack_offset); + else + static_branch_disable(&randomize_kstack_offset); + return 0; +} +early_param("randomize_kstack_offset", early_randomize_kstack_offset); +#endif + void __init __weak arch_call_rest_init(void) { rest_init(); From patchwork Mon Jun 22 19:31:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11618853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 956BD912 for ; Mon, 22 Jun 2020 19:34:05 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6CACC20776 for ; Mon, 22 Jun 2020 19:34:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="f/7bNQCN"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="YJuQG2GV"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="J/uzTV9o" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6CACC20776 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=JuSSTtRY+Cey41/ge/ykXWTsuOw6L6S6VZdRrRcON7c=; b=f/7bNQCNahbrG4j1vHD3gT+gy oDNBWRBHvB8dH25ewVhsJVMd7/F3yv507yIq3XMdDjXs2/AuMPcu5+gWy1PeKsawEXUTW61yXaQuG joYWB/9viQxnlRA913udkOTlTlclHBooaQjqFZ7hy8meEKt9eKlGQzmDxsXim4ihuNcXisqoXcX0M 8IwRMnUqz1IdtkaOyx98kqiVGvxV6HHLdWk1vW0pK1YYFaxq4JaySfhtBaBKRuU8Vm4HWpjLP0Qq4 aU+ZrIN+F3xpWswujxNso6GzrdaRkXmVdfKlSD7e9IYtXJqO62+Osa8rCILbQSliVTSQuwxNv8HMZ o1gOpyd0g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAp-00023v-Mp; Mon, 22 Jun 2020 19:32:07 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAg-00021I-Vy for linux-arm-kernel@merlin.infradead.org; Mon, 22 Jun 2020 19:31:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LkjcF2W3iR+CPlZbMflQ6gTadXRXXr87sQKE5d5Gz6g=; b=YJuQG2GVNH9tCj2ZTmqUyMWyEu lQo7R2eLYoewBsrmsacoq8fuWzDFq93YTrmSG9XmlA+TEwOnF7IQ19VK5XohLW9i16GdI10vgUcIP 0lMDOQM0fIr1lfPepyrJZM+QoZWPFf9/fgwV236ue7XzB3eWXfpdhrU12axjQHCifkoJX83E95Wtn PBpPTNG9WAYiYRLrbJ2XZxXBY56rfXBl/rhFmC6QVJowSIdaPdFFDJy7U4FEDRKfgMqIM61plhiEL 2mZmCQt7yCuRWTJyn3jBpGO3BmELxj5H2tHDoHvGGFaYwVEl2nV0bbJ0AyT+Qh9vacSUXwSp3vKmh 2DZ6LmqQ==; Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAc-0002TE-AV for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 19:31:57 +0000 Received: by mail-pl1-x641.google.com with SMTP id k6so7992902pll.9 for ; Mon, 22 Jun 2020 12:31:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LkjcF2W3iR+CPlZbMflQ6gTadXRXXr87sQKE5d5Gz6g=; b=J/uzTV9owlvRdao+cxwp8Vslq/cFyZE3Lw8w6UUjzYWCr5LjrjVVNFSCVkinxX4Y2H kx/K7FzydtI/KfWHnOel3UOO4tgm0ampnIQbO/M+28yKFT7/EZw/h52Jk9ZvEqY4xdyo Gk7/un3Abi/4AXl0tPJ/akjqNspTqiopNeHAw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LkjcF2W3iR+CPlZbMflQ6gTadXRXXr87sQKE5d5Gz6g=; b=BTO2SH9wM8hutltea1wemT8LhLu/rFT74DKAYkAa0abMH4x7EMWmlkovLP/sM6BfK+ fomNKk1jf88TA7BRMEiQ1H2QfhMJH0BajSSHGiVOWGu08z/V6cV1mIsVgAEvO7e/1AEM a79u3Tf7X6t6zOPwMi6MdETa58SGXjDTgMq99y2D/tttOsRs+4EnnrHzO0KqPzy+NMef BV240qaCih5l0JNoi2WCQzXoXK9eiexaudqjHJgPEGigaR8/EpRtbCalCjZtV39HaLPf jS3zGH1ZNtEqp/F375UFJWzpNPaQA0rVivcZTmO7XJhWgUYnbywFB+R5sTy/Cw6hHHCX Mjjw== X-Gm-Message-State: AOAM532yTrSfR11c3AivOvPH5R50v4c0FbmFkAYbmcvqjdbl0mCV6zXh MputruIveasstZdovgzlYKMN4A== X-Google-Smtp-Source: ABdhPJzsD2uLlmJ7YMnz76yJmcuXkfmLp3DfOApUsk4PZt0cKKx5QBWDZp5Da97pcoyR3Ss/b2BgBg== X-Received: by 2002:a17:902:8bc7:: with SMTP id r7mr7968646plo.174.1592854311241; Mon, 22 Jun 2020 12:31:51 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id i196sm11565386pgc.55.2020.06.22.12.31.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:31:50 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v4 4/5] x86/entry: Enable random_kstack_offset support Date: Mon, 22 Jun 2020 12:31:45 -0700 Message-Id: <20200622193146.2985288-5-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200622193146.2985288-1-keescook@chromium.org> References: <20200622193146.2985288-1-keescook@chromium.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200622_203155_390250_1AAD6236 X-CRM114-Status: GOOD ( 15.17 ) X-Spam-Score: -2.1 (--) X-Spam-Report: SpamAssassin version 3.4.4 on casper.infradead.org summary: Content analysis details: (-2.1 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:641 listed in] [list.dnswl.org] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova , Alexander Popov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. Signed-off-by: Kees Cook --- arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 6a0cc524882d..57c2a458150e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -156,6 +156,7 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index bd3f14175193..681125bbf09a 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -26,6 +26,7 @@ #include #include #include +#include #ifdef CONFIG_XEN_PV #include @@ -240,6 +241,13 @@ static void __prepare_exit_to_usermode(struct pt_regs *regs) lockdep_assert_irqs_disabled(); lockdep_sys_exit(); + /* + * x86_64 stack alignment means 3 bits are ignored, so keep + * the top 5 bits. x86_32 needs only 2 bits of alignment, so + * the top 6 bits will be used. + */ + choose_random_kstack_offset(rdtsc() & 0xFF); + cached_flags = READ_ONCE(ti->flags); if (unlikely(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) @@ -346,6 +354,7 @@ __visible noinstr void do_syscall_64(unsigned long nr, struct pt_regs *regs) { struct thread_info *ti; + add_random_kstack_offset(); enter_from_user_mode(); instrumentation_begin(); @@ -409,6 +418,7 @@ static void do_syscall_32_irqs_on(struct pt_regs *regs) /* Handles int $0x80 */ __visible noinstr void do_int80_syscall_32(struct pt_regs *regs) { + add_random_kstack_offset(); enter_from_user_mode(); instrumentation_begin(); @@ -467,6 +477,7 @@ __visible noinstr long do_fast_syscall_32(struct pt_regs *regs) */ regs->ip = landing_pad; + add_random_kstack_offset(); enter_from_user_mode(); instrumentation_begin(); From patchwork Mon Jun 22 19:31:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 11618851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 722D014B7 for ; Mon, 22 Jun 2020 19:34:03 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4C0F0207DD for ; Mon, 22 Jun 2020 19:34:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="tsNJxQYO"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BKy23l2w" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C0F0207DD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=8S9p7+TiqhGQt1M7KxWgwEbXi5T8xB2vmMDABhEWnfc=; b=tsNJxQYOLa5x8ObYDnPNP6hB7 kLNr67dz1DoDeF1PN7QaoRZZIz/DBXDHBBjNm2TFTbviQuuHLWAgHRbkM4pVwgg/h+X1pDPTABU/N Tm7XyxeDehCNSVyt4Z0onN+RQsHjByuW03N9fJ3XaFLpFtokY17ciw3HOso6jqyZydS/mzvqCue22 kbtHDIfS9yhJ99pe6FY+rMlRNXZ7+WLkm/CS+Pxs0Vve/NeHdxVi7nqFvVFOOQsaYv7vu/tHYtGzE I7JuzQ/ptSorBL6Vbq2UG8uWhMZdg6Wc48la9Y4VkC4GDmzQmvGOzErAjPQKgJ2mUFHX4Gyw1Mz8i 9WHqQcS4g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAw-00026N-So; Mon, 22 Jun 2020 19:32:14 +0000 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnSAf-00020Y-5j for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 19:32:03 +0000 Received: by mail-pf1-x444.google.com with SMTP id x22so8866011pfn.3 for ; Mon, 22 Jun 2020 12:31:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zcPQ3teAeONtZCSjASMvWQH+0FGked91D84DWb3cnhA=; b=BKy23l2w1m43ZpBXQM2fDRryt6KkrHZIrILWSDueLTLg7JkO/xXIBdrpxBtpXqG8Eo C+whcf+rw091oqysj+R7yNmqCI48ZtM7rhgZGnobI7SE5xNIel15JCPoNXr8RLA/8gKA FUXw0i2C4M8kbA9Vh9+15Qvp6mVOpj0mucLag= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zcPQ3teAeONtZCSjASMvWQH+0FGked91D84DWb3cnhA=; b=rjniCfIdcMyu9iRv3pHRyfpnvhy6zZUAEMmdO6syZv2IS5FO1rnLw/JPFTBxfk6Dqh 1ooRjLGpUANK8Qsb0FOyvx7/cqhjnhLa4ePcWdKgdow47E/oCnFxM3OdoXKX5XIKUKvw yGZSpvden9KJgq4/MLki+0Qj5BN2iYgKaQyoGiFO255MMNc1e5mDYkIn0Ia+tgii/Acs z0FaPqMfFqhHpXfbdu1sEKaQohtSVH1nxJ9XNVD0248zEf7emGfhoy7pgN9/fZ1TsJ7V VTR8BQp6aq1PIdanyhTPR2fA8EhXtPJ5uY9sWHA1SIs/7nOZx44J5WgrlFqp4eVA3TC0 DY/A== X-Gm-Message-State: AOAM530pfxXRNSDCShlCmMC0fXZWYSePR9xzAFJB3JUkco74kDntm4cc VugknlDoG5qCCgRNshAqJ2vJhQ== X-Google-Smtp-Source: ABdhPJwITrbj7E83VXsf9fWrxu//1aHMGodlvPkjapka8zNgM+eZ+A/1KAwdust7T+yhFUZyNvTBXw== X-Received: by 2002:a63:3f42:: with SMTP id m63mr14501829pga.310.1592854314990; Mon, 22 Jun 2020 12:31:54 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id n189sm14916252pfn.108.2020.06.22.12.31.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:31:54 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Subject: [PATCH v4 5/5] arm64: entry: Enable random_kstack_offset support Date: Mon, 22 Jun 2020 12:31:46 -0700 Message-Id: <20200622193146.2985288-6-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200622193146.2985288-1-keescook@chromium.org> References: <20200622193146.2985288-1-keescook@chromium.org> MIME-Version: 1.0 X-Spam-Note: CRM114 invocation failed X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:444 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Kees Cook , Jann Horn , Ard Biesheuvel , Peter Zijlstra , Catalin Marinas , x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexander Potapenko , linux-arm-kernel@lists.infradead.org, Andy Lutomirski , kernel-hardening@lists.openwall.com, Will Deacon , Elena Reshetova , Alexander Popov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. In order to avoid unconditional stack canaries on syscall entry, also downgrade from -fstack-protector-strong to -fstack-protector to avoid triggering checks due to alloca(). Examining the resulting syscall.o, sees no changes in canary coverage (none before, none now). Signed-off-by: Kees Cook --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/Makefile | 5 +++++ arch/arm64/kernel/syscall.c | 10 ++++++++++ 3 files changed, 16 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a4a094bedcb2..2902e5316e1a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -135,6 +135,7 @@ config ARM64 select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT select HAVE_ARCH_PREL32_RELOCATIONS + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_STACKLEAK select HAVE_ARCH_THREAD_STRUCT_WHITELIST diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 151f28521f1e..39fc23d3770b 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -11,6 +11,11 @@ CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) +# Downgrade to -fstack-protector to avoid triggering unneeded stack canary +# checks due to randomize_kstack_offset. +CFLAGS_REMOVE_syscall.o += -fstack-protector-strong +CFLAGS_syscall.o += $(subst -fstack-protector-strong,-fstack-protector,$(filter -fstack-protector-strong,$(KBUILD_CFLAGS))) + # Object file lists. obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ entry-common.o entry-fpsimd.o process.o ptrace.o \ diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index 5f5b868292f5..00d3c84db9cd 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -42,6 +43,8 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, { long ret; + add_random_kstack_offset(); + if (scno < sc_nr) { syscall_fn_t syscall_fn; syscall_fn = syscall_table[array_index_nospec(scno, sc_nr)]; @@ -51,6 +54,13 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, } regs->regs[0] = ret; + + /* + * Since the compiler chooses a 4 bit alignment for the stack, + * let's save one additional bit (9 total), which gets us up + * near 5 bits of entropy. + */ + choose_random_kstack_offset(get_random_int() & 0x1FF); } static inline bool has_syscall_work(unsigned long flags)