From patchwork Tue Mar 9 21:42:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A0B1C433DB for ; Tue, 9 Mar 2021 21:45:39 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E6FD064FC6 for ; Tue, 9 Mar 2021 21:45:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E6FD064FC6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0gGDXzccB8u+gNWyG37Ie5jK419NcpbqRfH+TtReCTM=; b=m0F1xfOqRBrs5fE5WPrRhtKGL nVXuFA1L05BM9jkins4em62cnnKsxGjz/cnC5VrvNS2D4dOTcT84VPX9PrfJg3PHf6ey9WXB/8Rvz 7azm/pGvmX7dEBcw2Z3QC/gN5UNv0i2XdyntLWfPxkH9wlW2ub8gW23I2y76byF1USc2ho3SZIF+e ukgctPaGF1DABcfnXxiYtChtfcpkADaB5tjrq3RFe85DX+X4JRfoQ7NIczM0o1iTeSlVcdUqWAwhP n+qZZfZr+bL5uFJXWBpcMPFf7QKC7YacBDBM2xg33qQMTN4HK9IwCnNcEgcHHsb32+TWRyzZpm7Db yWJMQ6dfA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk95-005UAO-Pk; Tue, 09 Mar 2021 21:44:04 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8M-005Two-4w for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:22 +0000 Received: by mail-pl1-x62e.google.com with SMTP id a8so798299plp.13 for ; Tue, 09 Mar 2021 13:43:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uJhcYxfyQggjoFPfbV3IBwgqNX6derqJ/0OIfES3kng=; b=jL8KGxLZpq6kPeT9F+1qoL3DOavcSq+bJ2zvXDYPWXl+6YiuZBCwwRfCE7vfScpXYo JLjeD2AAKSyeha5oGCTqtO44VFZbSaDU8csNlwFLUVsaobMtQxYqvBKu4ZTMPr3ykAR6 o5hUKHMrDsMF1SIVJQ8Kd1W1dLLgNv1pQqr3M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uJhcYxfyQggjoFPfbV3IBwgqNX6derqJ/0OIfES3kng=; b=j5HNf7QjhjkCInZMByBXjZi5FSv4/LIMwffGw64gEMh3vD2hhrfnbERjfBVP6sY+eM 9j7UZ6btSlwMgvqOcZuEim+N3Lz6rRtaQWKdeLzj4vuNl5/V8fSknwPDT4JWhPaFdwjY K7hke6h0+dBngdXqCX+7J4u8LVHQy6EviB7p57b3Et69p5UwUPl0oTRxJm6etEEygzVU r1i3DnZ4q2SDCHEKT+Od8t4zvIUWGtT1uuqQ8uLGkcurh51nZuMOgOVIlI1UFs7X0iBQ bvn9W6aJffXkmkcHUJfHS8MMaBDJ+DYE/p42NZ18LQk8tDHeNUZZ3FIKtDCQlY88d5Q9 fBjw== X-Gm-Message-State: AOAM532ua4j51oo8GurPq8GNpXYyPC+So6mPLI9JwCdx5JCIikwQTjLj NR1pQXbktm9Lf3yypHWJ+dGE6Q== X-Google-Smtp-Source: ABdhPJzbrDwJvtujX+Yvv98LI2Jlt3lOh8R9CznfILmERfEu/fMLAFatr8X7tXLsHQfwfaOl+hu6bQ== X-Received: by 2002:a17:902:ec83:b029:e3:ec1f:9def with SMTP id x3-20020a170902ec83b02900e3ec1f9defmr107248plg.11.1615326189697; Tue, 09 Mar 2021 13:43:09 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id d20sm3872315pjv.47.2021.03.09.13.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:09 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , stable@vger.kernel.org, Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 1/7] mm: Restore init_on_* static branch defaults Date: Tue, 9 Mar 2021 13:42:55 -0800 Message-Id: <20210309214301.678739-2-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=28f3f839c24b05670130f4ff25d00e52971fd524; i=F1m1wEnux/IiRlF/sb2Zhr2mAPnBBAdImHuXCXK/7kg=; m=obdLQlMNg29JMC3GLL05bs7+HEYTRXfdcOPcqLYs1G0=; p=5DbOxVDVK+l3oSqEiw8KVTN/ODiEU3l6O/gw4X79jro= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+MACgkQiXL039xtwCZfHA//cAa PLXroyUxGbCfvAX2WTtwU6C5LPDtgM2cyDkhcMFkbY7jpjetZH58AgmsDVICQO5Gr1o6pcLanTaCu g9wn5kl+oYBuX+AM6N/RBjx2uuFb9H8vxS1sbup+S68TfmMPoa+8iLJYnDdQ5FeUCLzgIgdIoWyGC 1ebqyiytd8kEuZ54soVilY8JhdJ9Uyy2lj2CNuLncTdTZpB30J86I1fpIp8kG/1EqzwfOsrfK5WNE PlLp/tQYnPZw2LAxJg4fxq1OPvdXYXjOY6kVcx/tCaY3k4sEZjf+CA+N5ubqNZNQrcPo6yoUCuP34 iJpjSPR6EpsPimnEBVqI5dOnKcRbrQO/B96RbgbSqdGJckEiZk+/3x98oHJ7x0ZtQW/IDK82joRRf ZufV3KvwN1iLF2M0I1POqcKAk+9OF8mG/RA6pVK16vJmCA1iHTfLbz2JDUfWSu5rhFOYPCYs806V7 K8BpUEJz//Q38t6OXtzd/NuChkoMeEokmktZlFzmzLlk9qbAIWUrFHp8ECIbjYPNmaQ16YtU79tl6 qyAIlTkn5odVmnJxSTIKCZr7RwyszaDRBqPdPFSymmS/YdTKm+85L4N+AFtvX4V0NSjnSsYd5F0K3 q/LmHpjXj0aH6bVSWXXL2ifyidh60bHGGtZiwcWsIgkXi2mW0iYpkirvlgZc3gz4= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214320_109447_ADEA286D X-CRM114-Status: GOOD ( 13.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Choosing the initial state of static branches changes the assembly layout (if the condition is expected to be likely, inline, or unlikely, out of line via a jump). The _TRUE/_FALSE defines for CONFIG_INIT_ON_*_DEFAULT_ON were accidentally removed. These need to stay so that the CONFIG controls the pessimization of the resulting static branch NOP/JMP locations. Fixes: 04013513cc84 ("mm, page_alloc: do not rely on the order of page_poison and init_on_alloc/free parameters") Cc: stable@vger.kernel.org Signed-off-by: Kees Cook --- include/linux/mm.h | 8 ++++++++ mm/page_alloc.c | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 77e64e3eac80..b3317d91ee8e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2871,7 +2871,11 @@ static inline void kernel_poison_pages(struct page *page, int numpages) { } static inline void kernel_unpoison_pages(struct page *page, int numpages) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc)) @@ -2879,7 +2883,11 @@ static inline bool want_init_on_alloc(gfp_t flags) return flags & __GFP_ZERO; } +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif static inline bool want_init_on_free(void) { return static_branch_unlikely(&init_on_free); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3e4b29ee2b1e..f2d474a844cf 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -167,10 +167,18 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif EXPORT_SYMBOL(init_on_alloc); +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif EXPORT_SYMBOL(init_on_free); static bool _init_on_alloc_enabled_early __read_mostly From patchwork Tue Mar 9 21:42:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26AC2C433DB for ; Tue, 9 Mar 2021 21:45:11 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A3E6564FCB for ; Tue, 9 Mar 2021 21:45:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A3E6564FCB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LXdm8+7uqwTVYwAUO5hX/EEEcKwNUrbFemW60KHLpS8=; b=kH/pGFPmJpGL7FXzaFpBCtzfu qGpViMeo2i+6ACMUkZJz80ckxU9T7Xbavs5tJzRgliSFggnObJySGEfqccBp0zTIf7wr+ijrkm90v 4HJ+vYrQooEJ30wFngv2EK3r08I1UGIVecdzhbfNItxTr7CTlHyI3zSEsY8WKqCCh0IqNv6cgfAT/ oCwgQalWAmJv7PZEv2/f5fP0y9eWwvOwnndS0T7T4FjO4NfNASWQkFxRULzVYsNzjmVjBYNQuGIAs uuO2nGTKWUcEryb5PxkaZe8jSLcBrNuWfKHfbuUl+tu2Nn+hDFvwKvB6HfMYij1uCTLL0R/8G0/DA q9eJ2DEXw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8f-005U5n-Uo; Tue, 09 Mar 2021 21:43:38 +0000 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8L-005Tww-3I for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:20 +0000 Received: by mail-pf1-x433.google.com with SMTP id t29so10228253pfg.11 for ; Tue, 09 Mar 2021 13:43:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8B+TI2ljrjIAGoG2RdRj4gE9w9WbKxchx8DXaJn5Ui8=; b=B+YVHQykoUKo/84JBRqf2taJxNkvoR3h/xHiwuWVZPWtvPUuIKHLZrmpSpgh/Kjsr+ ZLGPtMIVy5G9SB0bStvEOZ5wssR0qJGsKIUTWMCiMDchjrHdU4QYbm/+AP7atg6dqz8I SNULNTYBNdoInlpqHUsBUJuHT5/aZTEFeNGzA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8B+TI2ljrjIAGoG2RdRj4gE9w9WbKxchx8DXaJn5Ui8=; b=ARqRya3YXTfcrCVElocAoV+rZkgbi0ALNji1j7JizzI7SWf9IY14JpQxLdIY/Le4u2 Yu76+/3udK1hmkIHNhH5qm7Zmw1FDWxRe5IBkvIVauiSTRn9+3Q6zGJqJ7rSlRCYFgME fH/eCdfHKswpQ7ZLBfLGLjakJOFQrGnD7cDGtpButbLJ61dCopUAUBOtGCaSIWmZmLQE EXv6zTXSTnXHVghpKbM0wf3nzRlY5cl/tNhVPCruf2T999tJbmeFWnZ6sWbl2ch/bul3 o6Wq9OiG4Nl2gCadP7KVLI1gREdKWhAbOgGbxfmSA52yYYqJPTXVX5nzPRB51RS2KUOh E6EA== X-Gm-Message-State: AOAM531bLB7AhHALI9DBB1dUNe5LcRs9Fim1l32Dbm+PGNkIw4PFF1cw BE9cziO2EVUF/lIM5H3XfHlXnA== X-Google-Smtp-Source: ABdhPJwDEZt4HsJYCPAO/hKVfuBRLIAx9UGXKG+K9UyVuDvSK9N5uemw3WHAwwMJqmSvMAkwDOlEtg== X-Received: by 2002:a63:db57:: with SMTP id x23mr10704884pgi.432.1615326191286; Tue, 09 Mar 2021 13:43:11 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id il6sm3550028pjb.56.2021.03.09.13.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:09 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Peter Zijlstra , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 2/7] jump_label: Provide CONFIG-driven build state defaults Date: Tue, 9 Mar 2021 13:42:56 -0800 Message-Id: <20210309214301.678739-3-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=73472045d5d9db41a0abdff057f03f8d5fac6646; i=Vishx6UyAXwYzcnoSyP+eBB3iQyx+/i5smsbQfc0cnA=; m=jNlGnJHaf6m7Hp53arzPhV1aLM4PJz/v+5ZYK3JqA/s=; p=ugkbC9pkmjvHuu7AHPV/hFDaWx06QijpAk87L8pMr40= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+MACgkQiXL039xtwCa2UxAAgjF MUwiQyPV2kKbnkLNuK+WcErNu8CNPTgLa8SZoTRxE2MkJJ8aHZt0cV/ufuXUmiYRuF34Dz8SL7+dE cRM5gf1jBFvD1eGwjIeMhgnDK6h4D2KWWbqf+fiSP0GyiBrgZ9xiiwGNui20FAjEIdEyHnu78hpIh 40hVgGFfRw8rrhX8hVMJGuhGefIFo6p56PRMUYJrypXEkVtn4rX4JCXSk0AzyCodpD0BkKKZFqpml mVIhMOwLE/AGMavf3mSWnEGu1V/rolBLokPg+fnGjEEtCYqdf+4poo1BQ5bDsxRFG4S8heHYN/OrW GQZdygkyKFwdPkuWH/pYql3gPWrH4iaUX/Za+JTNJUyS14yJTVIME8gaBJyt3Cp0qNRtOlq7PIvG6 iEv7fYs1XeM2C8fC9lZCHZDTpG3AExLifbA7M2CKPylraihXTJqkkATUePskVY9rMGktiuGECsHFQ zkRurZDp4GoyT/J+FzpXTAGHjMIll+hL/VE9awpXfqYsVgpfhB2lI5aO+1HpcKI5cCirFxc5nTZXb gqXwcS3HmcUDm4pSYqyJLmhIrR4+qP5TSYuZjMTUKP8V7kXlHXmBTrXU9+d5RptapTsAxFn4fwXj9 YmvarOr6C3Bykxp7PL4bm7pvHpPGYy1zZ5VDOstYKSa2ZJXl63cwLTiv2PBiDlfI= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214318_300764_638EB008 X-CRM114-Status: GOOD ( 15.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Choosing the initial state of static branches changes the assembly layout (if the condition is expected to be likely, inline, or unlikely, out of line via a jump). A few places in the kernel use (or could be using) a CONFIG to choose the default state, so provide the infrastructure to do this and convert the existing cases (init_on_alloc and init_on_free) to the new macros. Acked-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/lkml/20200324220641.GT2452@worktop.programming.kicks-ass.net/ Signed-off-by: Kees Cook --- include/linux/jump_label.h | 19 +++++++++++++++++++ include/linux/mm.h | 12 ++---------- mm/page_alloc.c | 12 ++---------- 3 files changed, 23 insertions(+), 20 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index d92691262f51..05f5554d860f 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -382,6 +382,21 @@ struct static_key_false { [0 ... (count) - 1] = STATIC_KEY_FALSE_INIT, \ } +#define _DEFINE_STATIC_KEY_1(name) DEFINE_STATIC_KEY_TRUE(name) +#define _DEFINE_STATIC_KEY_0(name) DEFINE_STATIC_KEY_FALSE(name) +#define DEFINE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_, IS_ENABLED(cfg))(name) + +#define _DEFINE_STATIC_KEY_RO_1(name) DEFINE_STATIC_KEY_TRUE_RO(name) +#define _DEFINE_STATIC_KEY_RO_0(name) DEFINE_STATIC_KEY_FALSE_RO(name) +#define DEFINE_STATIC_KEY_MAYBE_RO(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_RO_, IS_ENABLED(cfg))(name) + +#define _DECLARE_STATIC_KEY_1(name) DECLARE_STATIC_KEY_TRUE(name) +#define _DECLARE_STATIC_KEY_0(name) DECLARE_STATIC_KEY_FALSE(name) +#define DECLARE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DECLARE_STATIC_KEY_, IS_ENABLED(cfg))(name) + extern bool ____wrong_branch_error(void); #define static_key_enabled(x) \ @@ -482,6 +497,10 @@ extern bool ____wrong_branch_error(void); #endif /* CONFIG_JUMP_LABEL */ +#define static_branch_maybe(config, x) \ + (IS_ENABLED(config) ? static_branch_likely(x) \ + : static_branch_unlikely(x)) + /* * Advanced usage; refcount, branch is enabled when: count != 0 */ diff --git a/include/linux/mm.h b/include/linux/mm.h index b3317d91ee8e..bf341a9bfe46 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2871,11 +2871,7 @@ static inline void kernel_poison_pages(struct page *page, int numpages) { } static inline void kernel_unpoison_pages(struct page *page, int numpages) { } #endif -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_alloc); -#else -DECLARE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc)) @@ -2883,11 +2879,7 @@ static inline bool want_init_on_alloc(gfp_t flags) return flags & __GFP_ZERO; } -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_free); -#else -DECLARE_STATIC_KEY_FALSE(init_on_free); -#endif +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { return static_branch_unlikely(&init_on_free); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f2d474a844cf..267c04b8911d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -167,18 +167,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_alloc); -#else -DEFINE_STATIC_KEY_FALSE(init_on_alloc); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_free); -#else -DEFINE_STATIC_KEY_FALSE(init_on_free); -#endif +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static bool _init_on_alloc_enabled_early __read_mostly From patchwork Tue Mar 9 21:42:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F29EC433E0 for ; Tue, 9 Mar 2021 21:45:18 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C30E64FCE for ; Tue, 9 Mar 2021 21:45:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C30E64FCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=H+xp5J1DuEZFa8djWDsbv/8UyVA6tRa2EWL8cZ36agA=; b=SOFfzcbJmXTU9gKsu7aq87sbv od9oygEkRmYuqY2ujlSWC7o2/T3I3tbTuQSjP5xuguIex5WHUJrFYxLy6rNmyLmaAXJFpCJts3S+W e+qYb2imjNLdeO3JS4yfvZCxEMlwXTL1hzXDNV4EoJOKTRI6e7rgSnECik9AzF2G0moL5YKGtOcb3 WBYUhAqls382DbS8wU3o0zj3SoebGc11yGZzcN0aC5WuOUy/ghV+/2uHzsTHkmxErBya4oYX5ti8Q irTtpb6CelIeNwJSVFWByMSjQvrufsONbCIUYVgQVRyD9SPWfIzhvtuAFGZ6Qy2C81fIwKgncBWfF dBM61NujQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8n-005U6k-UJ; Tue, 09 Mar 2021 21:43:46 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8L-005Twr-Ih for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:22 +0000 Received: by mail-pg1-x532.google.com with SMTP id x29so9726507pgk.6 for ; Tue, 09 Mar 2021 13:43:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QchGfDa9Ang9HjVlU1RQNYIjoI3tEPOfhu9LIBI5D+M=; b=AAchEITQdw6OW0zYEsb+JMW1vwYgaTwlwzk7fiKypJWjhEQgoK92EG8oQj/4Tq88SF zwG5DspDifkgXW3HuJUgs4F3dISckAhmF8SPrLQMssyCJX95lcs5zaG5CrPPPGeb+HSO dWyEMq2T8jbT0sDAA79Mlwjm4ookDIWPgIkLI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QchGfDa9Ang9HjVlU1RQNYIjoI3tEPOfhu9LIBI5D+M=; b=kTWr9zJwlGNNx9ofA/Gg0s2sACnjUWYt/FquUZOPm/4GfnkYZwpFwNd+UaB83MQDmJ tqNwsPotkgqi4m+kWppw5+HuszaZlCnCXHh0ilTL21mcMiS72ZyrcpJbjr4ohHUk/+Rq yUTK6DE9HEnIIypQAWpeqYPZiURSWSi9wOFrpA6I/IxOn5Xy5L7SOQNf/hC8elIkbkyd Z8B7APmeyAQbqe2O5xLvk7vTMAcSDHNdVBvswOFeqG6J5PZNvyvscwhY3b1wijXMxL0X RjxTFsty/TLecXgTd7duAo8i7CyDobgqH9lhwtGynTWeUEL67p+LXH6LrPw047bLfIVH m1XA== X-Gm-Message-State: AOAM533s/2rXpXqD5bpJ8P89koQ0DSv32QZh0f7M3ePUTp8k7Jq+gE+U eF1eOkR+X7YwNyyD9Ofy8qPzSQ== X-Google-Smtp-Source: ABdhPJzcdsp6YYQ3RJHVjDIyR4wddA7Ee7emO+fBuKmY/97M5tY6wZi5jJxb+krjkrH7lCrJYsY4vg== X-Received: by 2002:a63:4241:: with SMTP id p62mr26586949pga.453.1615326189986; Tue, 09 Mar 2021 13:43:09 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id y8sm15163590pfe.36.2021.03.09.13.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:09 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Alexander Potapenko , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 3/7] init_on_alloc: Unpessimize default-on builds Date: Tue, 9 Mar 2021 13:42:57 -0800 Message-Id: <20210309214301.678739-4-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=0180538bd89f892d9a7cb5912f57764dd4574e64; i=FGm2eJrFhO36W3pokRKQXJ/1lCJEdw+Xji0AZWdKldw=; m=vArMoxGjWi+hJ/sQRlVkEVurD6o0k4s1dF+QIrjNXSc=; p=UIEnT5JyleN0fGLkjBMkQSBdyQITBdVfP87yTFfx7Kg= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+QACgkQiXL039xtwCZhGQ//cU9 87sq54cQfs8uAKxVAHV77yHdW6J6AftBwhrqAq+1M0C1IoOLVi+Uvz1cWTM7C6G672FEhYPM8aZvd 31/s3taFnCDyvmBhh9HUBBsYUk7Hhwi+V7r5EvVHAxBw1jyUByiPVHV6FWRcSDFOZHv9iKXNEFC7O iKSWVZ4V6X0u98fHeVq2MA3000qbRbBtef8DMgWd5rq3d+1IJQKEWRi/ws2e3bW82oHj1a7+KPOv4 X1J5M9A/OStaOcbwJ76LyVjtjizbvJMYsUDfIwUkH2Ouj84CYuWxK5AflU8Wf8690EOKQ6XGAvCXA 2tcD81q96YPjYRgGy1QoC4cqyGHQf52FCkGDjrZd4LIHJjPLxc8C7OAemJ8ofvZNIZDwRF9u07y4D qUKVUYEjWSwsP03gVJB7kc6gYALIB5AX3qNNQkMAGKgiyrLXML5eBay0b9+G6DrlXQO2uEwQ6ocni JoXsEVBoz0zZRetbT2/TXK6Nqk9DV4p8bkO7R26Ut6wAEn8jZSHgyBk3GrJg+6d1pOzsj761vvjya NK/0w2VQYOsrjdciOEQR/Xd2ZGCaFPfQY41b5nEjmBC+YS8l0GSHFqG1ce4mPDlQeuNmYOgu8+k/B 1hTaHBXrEPHJbO69gYEpIn/T7M1Tb4pYIlHnvmvDlMkPoUrznzLJonVQIk5rqeX8= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214319_639543_F598B431 X-CRM114-Status: GOOD ( 14.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Right now, the state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branch tests. Use the new jump_label macro to check CONFIG settings to default to the "expected" state, unpessimizes the resulting assembly code. Reviewed-by: Alexander Potapenko Link: https://lore.kernel.org/lkml/CAG_fn=X0DVwqLaHJTO6Jw7TGcMSm77GKHinrd0m_6y0SzWOrFA@mail.gmail.com/ Signed-off-by: Kees Cook --- include/linux/mm.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bf341a9bfe46..2ccd856ac0d1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2874,7 +2874,8 @@ static inline void kernel_unpoison_pages(struct page *page, int numpages) { } DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc)) + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc)) return true; return flags & __GFP_ZERO; } @@ -2882,7 +2883,8 @@ static inline bool want_init_on_alloc(gfp_t flags) DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free); + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free); } extern bool _debug_pagealloc_enabled_early; From patchwork Tue Mar 9 21:42:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E625C433DB for ; Tue, 9 Mar 2021 21:46:07 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6165364FCE for ; Tue, 9 Mar 2021 21:46:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6165364FCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cq6lSUnoJnchS5Ab/QSIaHtMErLFeWPuYWWvTIf5pdU=; b=GO5KOqrmzf502gLn3sO2sucxV fJr6WhwzQRGIHOFOrmldJ7BZqKaRZJVrW4/gKqXJMkm4nkLGgnhtloMnCvmPqQgs6e/FCIhpGjvq4 gWM2y05PKQWqH1Y+UWwwflrJRf6DzKFPrupxBIxCq02Wvzw7WT4xyTTd2+8u/YSDOJT1Q3DBspqEM KGbb/qClx0ISLbAsD/I7yjU9HMr9zCHpkleAP6WvERsj8Dj21cmufYFfGBUJrYgXEsuMiLEx9mxOL DPTCfDkZFXXsDfuUGEIU75Yw88y1ylttXaoj9fkLp1EJON1iY5Uf88C//zxVhs20f6MZr7LK4OFwg eWejpMnhw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk9W-005UJl-Sj; Tue, 09 Mar 2021 21:44:31 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8p-005U6s-5N for linux-arm-kernel@desiato.infradead.org; Tue, 09 Mar 2021 21:43:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8RR/gKm83SIV1ashXcsFL/lRx7PJhODu8ZSMn6SRrJQ=; b=mOMiqgTFqsEyyB0K4aQhrFgqvI HvSm4gankhu8qVOoD2xMcZhTTI0CAWU6xscIO9Rd0IPymtmAk3vsgJCUZQObpr+8gY9Zf9ZfLFieL SUtoMyHQM7l+aJMKflVFfGzQhLqKv15VqDp0sdDTau3eO19V/afXjYIwMJNWJU4scxokhhoy/oKEO ub0SdyDf9aJa+y7c1zFczH9cp5/jRxuhaj69och3Se0s1FjUEM2u/Z/5QnrBDvbYsLaLJVJAmmXTC RqHLvkGMSlAXR8ZyB6T11W0GHuv76IMBz69KQpXlqBPG/54vHXLxsPOd9r4zmlebLfVFpgsRGNpDX EMYmaK0g==; Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8U-001VRK-Q1 for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:37 +0000 Received: by mail-pl1-x629.google.com with SMTP id s7so7310140plg.5 for ; Tue, 09 Mar 2021 13:43:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8RR/gKm83SIV1ashXcsFL/lRx7PJhODu8ZSMn6SRrJQ=; b=Aa3aRs4OJPwblM5njJZhnEgcytwXidQByjYa9Ay6CSQY4T61mreRXkEIfUuJk4kW7V +SLSgbpCymc8I1YZCygo9ETUz9nK6SsozbQgbB/9c4IouQIbXj4TNJksDzJKw3cuGJl/ KrVvzIjE9iyZn7H8MJwRXZuySati6yJN0RMhI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8RR/gKm83SIV1ashXcsFL/lRx7PJhODu8ZSMn6SRrJQ=; b=eOmsdADQgCUkpuR9Kwknc2M8Ad3eifCEQKtnfbc+dGR2uSKrUoQTAWlDY7WkscQNyT VqTDBu35Ng2gr2SPhjprWrvXu/Z4S9nuYq2GBwp64Ccrv/iWJnOZikwimvJT85ZdmJqy kS46ZhSdoEc6/Mhe0EJywNofkzr2MWey2MDqSNM/aQfpcNWzHtaIeypUfYNKQaLhbnhb oVl0Gz1xP439jWqXgxD524OVQps+72b5slucSNgfKCzJac3RgTvZPLKCc645EP5jfmfS dVOr1GjHwtjUYPTimvmR6gaX5peblVcSXBuomiGUmc6v72oxbn+gdUF12Z1HPci3q4Ja 4QBw== X-Gm-Message-State: AOAM530ecNnOYe8SGnEaa7TduxgYmX/Z/oafMcESYodikmeiUgpJolQA ktpdoANVxV+6Pg7lh3of7GC/KA== X-Google-Smtp-Source: ABdhPJybP/9TJUD7D2FkuRFBUCYyX6YkmGRAq9gsWFwEjp+OFZfSYratNXF5gBKSQlHH8qnkVl7yvA== X-Received: by 2002:a17:90a:c249:: with SMTP id d9mr6828303pjx.104.1615326194695; Tue, 09 Mar 2021 13:43:14 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id w8sm13969566pgk.46.2021.03.09.13.43.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:11 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 4/7] stack: Optionally randomize kernel stack offset each syscall Date: Tue, 9 Mar 2021 13:42:58 -0800 Message-Id: <20210309214301.678739-5-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=4a65f719103e0800c3c67a72875df15eff769bc7; i=R0fcIKmEUh8zot7Dpk8JcJdIeXVfXkVsg5oQI7Df8iY=; m=MAP5vDRMDqR2Ss+UnPKNUGFW/8cuHmj3fGsgmrr5dWA=; p=FEZkCJfgrReOcdjubtlCDCLRXPUj5jJvx3leLrrKn8I= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+QACgkQiXL039xtwCYt/A//ZpW xIrPYTNqy9sJdKRbFSzhEskkhjw9aomRWQn/jYttnfZ4rtdHNQHHpeclAZFgY22YwhHqtQGw+n5hO XysR3g3QUyJDpHmdk2cOC7f9hQGr8nZmLutiw6eJshqxWlcLeK1kkXdvy09dASoExLEaO7WGVI2fV AuEg1C/N/DQSBcPG4F3RnjLvrhO4mKRAAMh5vQI16POw2F4DMtdDt6D4i59WyMf9b/18S5XuJacEz g0KJXCxHkLhm0t749F4I9OLhlWMVrev6z84lLnIEvrtGWIFDph68bz1LcwARlpBZQPmw2hQZWNpNj qyAobPazh6XAZLnWFjnhf6HtzR4I9a6efBEGig5QgQLFAoEavmWP6frgRESxXVZCTuaxPWqFU8UHc LS1mdbmBh+VepFNQ3ILtuZfZ4n0Kb9ri0jmjsxtOHU8vf5bphCXY7kstoSf0CjCvnn12SfPYBFYxU FgCatXZ1IfFzOpFD5jmqswppdv4BZkm8Lpgd7j1HhqMx785XkbkjHxjN9+Fe6VfjDiUtMa8TduRed R/TSmTcMzP6YBxXs+f/2smfORgho7h6a9mn6WsJ6n9yYPWATd3duVP+Hotd7wghl3DNlfcKbop1h3 NBc6ALYGrdHlh5oQWnmmmhR6kTp+a7nZJrAsgkEy1JY/bSQqZMlPcRUWLVydNK8g= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214328_054702_91EFBB3E X-CRM114-Status: GOOD ( 35.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This provides the ability for architectures to enable kernel stack base address offset randomization. This feature is controlled by the boot param "randomize_kstack_offset=on/off", with its default value set by CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. This feature is based on the original idea from the last public release of PaX's RANDKSTACK feature: https://pax.grsecurity.net/docs/randkstack.txt All the credit for the original idea goes to the PaX team. Note that the design and implementation of this upstream randomize_kstack_offset feature differs greatly from the RANDKSTACK feature (see below). Reasoning for the feature: This feature aims to make harder the various stack-based attacks that rely on deterministic stack structure. We have had many such attacks in past (just to name few): https://jon.oberheide.org/files/infiltrate12-thestackisback.pdf https://jon.oberheide.org/files/stackjacking-infiltrate11.pdf https://googleprojectzero.blogspot.com/2016/06/exploiting-recursion-in-linux-kernel_20.html As Linux kernel stack protections have been constantly improving (vmap-based stack allocation with guard pages, removal of thread_info, STACKLEAK), attackers have had to find new ways for their exploits to work. They have done so, continuing to rely on the kernel's stack determinism, in situations where VMAP_STACK and THREAD_INFO_IN_TASK_STRUCT were not relevant. For example, the following recent attacks would have been hampered if the stack offset was non-deterministic between syscalls: https://repositorio-aberto.up.pt/bitstream/10216/125357/2/374717.pdf (page 70: targeting the pt_regs copy with linear stack overflow) https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html (leaked stack address from one syscall as a target during next syscall) The main idea is that since the stack offset is randomized on each system call, it is harder for an attack to reliably land in any particular place on the thread stack, even with address exposures, as the stack base will change on the next syscall. Also, since randomization is performed after placing pt_regs, the ptrace-based approach[1] to discover the randomized offset during a long-running syscall should not be possible. Design description: During most of the kernel's execution, it runs on the "thread stack", which is pretty deterministic in its structure: it is fixed in size, and on every entry from userspace to kernel on a syscall the thread stack starts construction from an address fetched from the per-cpu cpu_current_top_of_stack variable. The first element to be pushed to the thread stack is the pt_regs struct that stores all required CPU registers and syscall parameters. Finally the specific syscall function is called, with the stack being used as the kernel executes the resulting request. The goal of randomize_kstack_offset feature is to add a random offset after the pt_regs has been pushed to the stack and before the rest of the thread stack is used during the syscall processing, and to change it every time a process issues a syscall. The source of randomness is currently architecture-defined (but x86 is using the low byte of rdtsc()). Future improvements for different entropy sources is possible, but out of scope for this patch. As suggested by Andy Lutomirski, the offset is added using alloca() and an empty asm() statement with an output constraint, since it avoid changes to assembly syscall entry code, to the unwinder, and provides correct stack alignment as defined by the compiler. In order to make this available by default with zero performance impact for those that don't want it, it is boot-time selectable with static branches. This way, if the overhead is not wanted, it can just be left turned off with no performance impact. The generated assembly for x86_64 with GCC looks like this: ... ffffffff81003977: 65 8b 05 02 ea 00 7f mov %gs:0x7f00ea02(%rip),%eax # 12380 ffffffff8100397e: 25 ff 03 00 00 and $0x3ff,%eax ffffffff81003983: 48 83 c0 0f add $0xf,%rax ffffffff81003987: 25 f8 07 00 00 and $0x7f8,%eax ffffffff8100398c: 48 29 c4 sub %rax,%rsp ffffffff8100398f: 48 8d 44 24 0f lea 0xf(%rsp),%rax ffffffff81003994: 48 83 e0 f0 and $0xfffffffffffffff0,%rax ... As a result of the above stack alignment, this patch introduces about 5 bits of randomness after pt_regs is spilled to the thread stack on x86_64, and 6 bits on x86_32 (since its has 1 fewer bit required for stack alignment). The amount of entropy could be adjusted based on how much of the stack space we wish to trade for security. My measure of syscall performance overhead (on x86_64): lmbench: /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_syscall -N 10000 null randomize_kstack_offset=y Simple syscall: 0.7082 microseconds randomize_kstack_offset=n Simple syscall: 0.7016 microseconds So, roughly 0.9% overhead growth for a no-op syscall, which is very manageable. And for people that don't want this, it's off by default. There are two gotchas with using the alloca() trick. First, compilers that have Stack Clash protection (-fstack-clash-protection) enabled by default (e.g. Ubuntu[3]) add pagesize stack probes to any dynamic stack allocations. While the randomization offset is always less than a page, the resulting assembly would still contain (unreachable!) probing routines, bloating the resulting assembly. To avoid this, -fno-stack-clash-protection is unconditionally added to the kernel Makefile since this is the only dynamic stack allocation in the kernel (now that VLAs have been removed) and it is provably safe from Stack Clash style attacks. The second gotcha with alloca() is a negative interaction with -fstack-protector*, in that it sees the alloca() as an array allocation, which triggers the unconditional addition of the stack canary function pre/post-amble which slows down syscalls regardless of the static branch. In order to avoid adding this unneeded check and its associated performance impact, architectures need to carefully remove uses of -fstack-protector-strong (or -fstack-protector) in the compilation units that use the add_random_kstack() macro and to audit the resulting stack mitigation coverage (to make sure no desired coverage disappears). No change is visible for this on x86 because the stack protector is already unconditionally disabled for the compilation unit, but the change is required on arm64. There is, unfortunately, no attribute that can be used to disable stack protector for specific functions. Comparison to PaX RANDKSTACK feature: The RANDKSTACK feature randomizes the location of the stack start (cpu_current_top_of_stack), i.e. including the location of pt_regs structure itself on the stack. Initially this patch followed the same approach, but during the recent discussions[2], it has been determined to be of a little value since, if ptrace functionality is available for an attacker, they can use PTRACE_PEEKUSR/PTRACE_POKEUSR to read/write different offsets in the pt_regs struct, observe the cache behavior of the pt_regs accesses, and figure out the random stack offset. Another difference is that the random offset is stored in a per-cpu variable, rather than having it be per-thread. As a result, these implementations differ a fair bit in their implementation details and results, though obviously the intent is similar. [1] https://lore.kernel.org/kernel-hardening/2236FBA76BA1254E88B949DDB74E612BA4BC57C1@IRSMSX102.ger.corp.intel.com/ [2] https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ [3] https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html Co-developed-by: Elena Reshetova Signed-off-by: Elena Reshetova Link: https://lore.kernel.org/r/20190415060918.3766-1-elena.reshetova@intel.com Signed-off-by: Kees Cook --- .../admin-guide/kernel-parameters.txt | 11 +++++ Makefile | 4 ++ arch/Kconfig | 23 ++++++++++ include/linux/randomize_kstack.h | 42 +++++++++++++++++++ init/main.c | 23 ++++++++++ 5 files changed, 103 insertions(+) create mode 100644 include/linux/randomize_kstack.h diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 04545725f187..bee8644a192e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4061,6 +4061,17 @@ fully seed the kernel's CRNG. Default is controlled by CONFIG_RANDOM_TRUST_CPU. + randomize_kstack_offset= + [KNL] Enable or disable kernel stack offset + randomization, which provides roughly 5 bits of + entropy, frustrating memory corruption attacks + that depend on stack address determinism or + cross-syscall address exposures. This is only + available on architectures that have defined + CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET. + Format: (1/Y/y=enable, 0/N/n=disable) + Default is CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. + ras=option[,option,...] [KNL] RAS-specific options cec_disable [X86] diff --git a/Makefile b/Makefile index 31dcdb3d61fa..8a959a264588 100644 --- a/Makefile +++ b/Makefile @@ -811,6 +811,10 @@ KBUILD_CFLAGS += -ftrivial-auto-var-init=zero KBUILD_CFLAGS += -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang endif +# While VLAs have been removed, GCC produces unreachable stack probes +# for the randomize_kstack_offset feature. Disable it for all compilers. +KBUILD_CFLAGS += $(call cc-option, -fno-stack-clash-protection) + DEBUG_CFLAGS := # Workaround for GCC versions < 5.0 diff --git a/arch/Kconfig b/arch/Kconfig index 2bb30673d8e6..4fe6b047fcbc 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1055,6 +1055,29 @@ config VMAP_STACK backing virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + def_bool n + help + An arch should select this symbol if it can support kernel stack + offset randomization with calls to add_random_kstack_offset() + during syscall entry and choose_random_kstack_offset() during + syscall exit. Careful removal of -fstack-protector-strong and + -fstack-protector should also be applied to the entry code and + closely examined, as the artificial stack bump looks like an array + to the compiler, so it will attempt to add canary checks regardless + of the static branch state. + +config RANDOMIZE_KSTACK_OFFSET_DEFAULT + bool "Randomize kernel stack offset on syscall entry" + depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + help + The kernel stack offset can be randomized (after pt_regs) by + roughly 5 bits of entropy, frustrating memory corruption + attacks that depend on stack address determinism or + cross-syscall address exposures. This feature is controlled + by kernel boot param "randomize_kstack_offset=on/off", and this + config chooses the default boot state. + config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kstack.h new file mode 100644 index 000000000000..c4701a39c21f --- /dev/null +++ b/include/linux/randomize_kstack.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_RANDOMIZE_KSTACK_H +#define _LINUX_RANDOMIZE_KSTACK_H + +#include +#include +#include + +DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DECLARE_PER_CPU(u32, kstack_offset); + +/* + * Do not use this anywhere else in the kernel. This is used here because + * it provides an arch-agnostic way to grow the stack with correct + * alignment. Also, since this use is being explicitly masked to a max of + * 10 bits, stack-clash style attacks are unlikely. For more details see + * "VLAs" in Documentation/process/deprecated.rst + * The asm statement is designed to convince the compiler to keep the + * allocation around even after "ptr" goes out of scope. + */ +void *__builtin_alloca(size_t size); + +#define add_random_kstack_offset() do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + u8 *ptr = __builtin_alloca(offset & 0x3FF); \ + asm volatile("" : "=m"(*ptr) :: "memory"); \ + } \ +} while (0) + +#define choose_random_kstack_offset(rand) do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + offset ^= (rand); \ + this_cpu_write(kstack_offset, offset); \ + } \ +} while (0) + +#endif diff --git a/init/main.c b/init/main.c index 53b278845b88..f498aac26e8c 100644 --- a/init/main.c +++ b/init/main.c @@ -844,6 +844,29 @@ static void __init mm_init(void) pti_init(); } +#ifdef CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET +DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DEFINE_PER_CPU(u32, kstack_offset); + +static int __init early_randomize_kstack_offset(char *buf) +{ + int ret; + bool bool_result; + + ret = kstrtobool(buf, &bool_result); + if (ret) + return ret; + + if (bool_result) + static_branch_enable(&randomize_kstack_offset); + else + static_branch_disable(&randomize_kstack_offset); + return 0; +} +early_param("randomize_kstack_offset", early_randomize_kstack_offset); +#endif + void __init __weak arch_call_rest_init(void) { rest_init(); From patchwork Tue Mar 9 21:42:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FDC0C433DB for ; Tue, 9 Mar 2021 21:45:13 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0559864FCE for ; Tue, 9 Mar 2021 21:45:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0559864FCE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RyROqPeD3DMGvyCWn1m9b0vfAX88EY6e81zqEbAncfg=; b=RtU9dHFFgbnzgs5sbmBeGWJ5i istoNXEkD1rQH0zazp6QLQzqScd3TzNW8wOuv9OL+Sf2sYRYfh0cy/cDH2aH6jls19q1o7r4lN0LC QO0+6AgWJC+3MN81Vao4zhVTr74J9UtE1o0Cwdxq6Bs/4iqtjJfPGqXfa1pRvnx1zZixVfY+1UBSh HIgt8wiPj181UK944wJrWZnt1CKwH7elIBh5Opim99UK5QdgCOQR3SG+I0R48yDpK8hZNy2FCI5VY RcZ88lSoxhIGRdLU+6DBxCg/4Eo0etEoI0zNaNzT8c8jJoB2l2ydkgnXG8jZQlTsioUAOGbGpG0rC pRqKWs/Ng==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8V-005U2w-La; Tue, 09 Mar 2021 21:43:28 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8L-005Tx4-1c for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:19 +0000 Received: by mail-pf1-x42c.google.com with SMTP id q204so10250729pfq.10 for ; Tue, 09 Mar 2021 13:43:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DcTnVacz4A0tmKEIIvO2n/EpgXlfdRl1ckINTgAR5oI=; b=g9mbVr6V7xJGugC2gPTw2AMctj391Z+XCiZu7k+SK//LiTMM+QBj3M/FHgckF+qI9K N7xliOCzmIGrsjVEKNOpDcoZWE/5lvFz/52as57ruksFXts4vF46jhV46Xp9QyBFJci2 1Uo819VKYBW9PfGRN67ylRVMUWziyI7JyosFQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DcTnVacz4A0tmKEIIvO2n/EpgXlfdRl1ckINTgAR5oI=; b=QaB96OUviBlB0pHaGUzU0JVQ8p7z4Z7X5ZwDZNakpUd+3zc/SloWLL32TiG99JxLpV lZYr/ko/9BO9lmx3g5kpXs7fkrbVWfYv8qmqayCP7s6llEGdwZ3S12p/LaYHgYtZUBli 3dX0Wn19xpFtFa+XLrmYOroUqzT22ail4HI510Hl5kxp5Ho0qHdtHr0+7RlF2we33+bo HHib2swBNk84o2sYPX4KLdRdc83tiq2gCCnfNY7OOHt9nWVfHlI4W3foMSD1fijVFnse 5Z6nmoxhm3sk97LH8pnBndcdWWDZ/JxsS99fDSaikB5JX1X0SFOgXL2BGpLg9wdRAUbN ID0A== X-Gm-Message-State: AOAM533A+gzUp+DOhJA7ABvfye4CS9wO96p6OwiTqppmGBuXIb8rVqvV h3zHjfzABjkYEm2LKrRxKOnd1g== X-Google-Smtp-Source: ABdhPJxgaMq60/8LT1KyAa101Khr91ahTpw6YQ74MnNFEX+siGaxShBO638RxRvX1u9rVkRXcE8twg== X-Received: by 2002:a05:6a00:2cd:b029:1f4:c3db:4191 with SMTP id b13-20020a056a0002cdb02901f4c3db4191mr3893pft.71.1615326192428; Tue, 09 Mar 2021 13:43:12 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id p17sm5917461pfn.62.2021.03.09.13.43.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:11 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 5/7] x86/entry: Enable random_kstack_offset support Date: Tue, 9 Mar 2021 13:42:59 -0800 Message-Id: <20210309214301.678739-6-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=1a3993b6371bcc7c9fa1576e563a724a70c25875; i=np7yed3mY+gWIWkexmB7CyDLgwsIh0xV2RGaksJc7tI=; m=cc9SLz8pm+mdXOKZLHtRjH7FITDzwHCOKmwFrX1p7io=; p=UyRR1nzRrbUXeUU5ICay1xt2wYZVgu7b/Eh0FjkN2Hc= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+QACgkQiXL039xtwCYLvQ//ZVV jKPhAQU2IVFJZBsGYqSYa3g8Q8uXNjqNouwYZkGdZqCp2yYRsOzmXFeH0qyZJScVL7wk5Ocm/QTFW OZermriCXaGfjKrBkgIjBKWkpovFITXMioqiiczOh7oKZdo8pNVZ1ORyZoNOeQS3VBXKjN+07PE3h dMMzX56qSTd9WKxNPLKAo2kcBNV1vyrAW8pz2WcZvQEqImlpKpR4XxS7IYdK3YivXvt98Ny1/RRJS RfubXI54V4KwvZGGc13jwp1UsZfgnRAcxEHTSz+jlpK5CyZkfqgBNOQBIJd/bmUi8iMIPRB1fkHMY vTp2Nn3+PBSlIpGQiDyMoHW03a92fqONW5g98Ci048GC5vwFsjQbWB66SuAg/sf5qWZRmLYhxaHqR 8l56FZXXqzdjtLYuQvLxARNqgiYjjON93W1qxsHO1MoNsGbTGYOwvdHF+/70kZjRRS+4a4z21qQcL Rxvaadi1lQYfNlwgu0D7FoJcOxT12AJ8O7U1wLiIWroeJVxpYfq3sdQ5UT7Bgw/+Myc/ElGZhVK7m L0S/dtlbsTNa2yVUxTFGp4+UXIaH3CYIsAlrK63FimB7nbe4+a0mzl5DhJ+6+Of1ZGbvMlJnGyFsL 3EmbxC4ZdezBe/5xGxUELY+eR9ndl+C3u2gZzoCCKkJL+vEtcYX35II8VFuA2ab4= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214317_679519_36ACD41C X-CRM114-Status: GOOD ( 17.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5-6 bits of entropy, depending on compiler and word size. Since the method of offsetting uses macros, this cannot live in the common entry code (the stack offset needs to be retained for the life of the syscall, which means it needs to happen at the actual entry point). Signed-off-by: Kees Cook --- arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 3 +++ arch/x86/include/asm/entry-common.h | 8 ++++++++ 3 files changed, 12 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879d398e..4b4ad8ec10d2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -165,6 +165,7 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index a2433ae8a65e..810983d7c26f 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -38,6 +38,7 @@ #ifdef CONFIG_X86_64 __visible noinstr void do_syscall_64(unsigned long nr, struct pt_regs *regs) { + add_random_kstack_offset(); nr = syscall_enter_from_user_mode(regs, nr); instrumentation_begin(); @@ -83,6 +84,7 @@ __visible noinstr void do_int80_syscall_32(struct pt_regs *regs) { unsigned int nr = syscall_32_enter(regs); + add_random_kstack_offset(); /* * Subtlety here: if ptrace pokes something larger than 2^32-1 into * orig_ax, the unsigned int return value truncates it. This may @@ -102,6 +104,7 @@ static noinstr bool __do_fast_syscall_32(struct pt_regs *regs) unsigned int nr = syscall_32_enter(regs); int res; + add_random_kstack_offset(); /* * This cannot use syscall_enter_from_user_mode() as it has to * fetch EBP before invoking any of the syscall entry work diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index 2b87b191b3b8..8e41566e154a 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -2,6 +2,7 @@ #ifndef _ASM_X86_ENTRY_COMMON_H #define _ASM_X86_ENTRY_COMMON_H +#include #include #include @@ -70,6 +71,13 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, */ current_thread_info()->status &= ~(TS_COMPAT | TS_I386_REGS_POKED); #endif + + /* + * x86_64 stack alignment means 3 bits are ignored, so keep + * the top 5 bits. x86_32 needs only 2 bits of alignment, so + * the top 6 bits will be used. + */ + choose_random_kstack_offset(rdtsc() & 0xFF); } #define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare From patchwork Tue Mar 9 21:43:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D74EC433DB for ; Tue, 9 Mar 2021 21:46:24 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 76DDA64FD1 for ; Tue, 9 Mar 2021 21:46:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76DDA64FD1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=S0tApvcGz0BPZalyNfyrQot0aZqj4AEzcL4QokvdNhw=; b=C9VroqxmiOLffFybjCFyqQRIg TloIlBBIuGYUa/tFmqlYb0ZiL3tZ3f02+eKG5AuZUP79A4QUqxw4iUwJ1WlOBvlLYFScXK0ivVKFK 7bBNzoS+QttR/kaBux2B9ZU8xL5xzpiGkm+VgBTa2pnMjeOT6MDZrCgiF6eziDVKZYqvu7rOeM4Hb YdY5s2jfSqCmEFCjFKS46o/Nt1dloLDPIAZH47feL8DSM7DnKbt3i1YVPuLpkl7JbbTxaXxLsdaSO FIVyYXmVf88cqFSRJPozSHCqu/pqA2JKP2MvV519fhYz0ht4aXmgc/AucOsoQ4OsiLraT2ly0GDYZ /ReGUfM5g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk9k-005UPo-He; Tue, 09 Mar 2021 21:44:45 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8w-005U7z-E9 for linux-arm-kernel@desiato.infradead.org; Tue, 09 Mar 2021 21:43:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=yYr9wVtQr+Yv2YCxxAWFJhlutYOKWxR3zpP8Meqzwc0=; b=Mj5IocliC3kueZk6X0LGX9xEZB UWlekqZFTX8axjujKVYr/B+TVvxPVNWGNszgyAlBYwohdK14iX2wvxErtpIEJDoGODvVdvZkmSBDq XGI11SVItYLowH8UmoDgYc+G/edUIaqTpcSqHGlK0AV30fFj9YqVdm1Sk4PZMEQSPpJqKAdpjVBwp +XSiR+oKfRr3JO226wOhOrh9Mj72Sm8VosNGGRL7KPNz4+VXrbJ7E/Xo4gspdFqP26yVcbfoqLunE 3I1g6EEE4k2DpbAOUD1/BwBo2SxEiQmrn/6lyGd/VPlxqPk1msjKIq3uFwie8y4tmcfcH/YPgY77/ 4DptEPdQ==; Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8V-001VQd-O5 for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:44 +0000 Received: by mail-pg1-x531.google.com with SMTP id x29so9726626pgk.6 for ; Tue, 09 Mar 2021 13:43:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yYr9wVtQr+Yv2YCxxAWFJhlutYOKWxR3zpP8Meqzwc0=; b=QfuDisPe9e6LA5+G0OqhJOKbJhyOfYMgFYd71oYjyW/qjnQRskxH2DwSwWa+zduR8z WQIq/CMZhY4sesmNBwH3ZsjovTqvPGWkSa5KYVH6qBsI8Qa7U87NKh3nubJFEJT8vq1A thA54toHbJNY99F/Bh83DjOw1zoidC/seKkGg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yYr9wVtQr+Yv2YCxxAWFJhlutYOKWxR3zpP8Meqzwc0=; b=iQPx+LJuMh6en6Bmz4bSjiYyQ+FG06Q7SPM5GC1frWkUndTDr6RMDdHEbHx1opUPxx xKRo84NRkvhydUXRMm7nhv4xR6uz/EomlZ5cakymqG5U+5fHJYGVau1OReNASFFYhFBh liZLzdLK0NCN8rxcU9PbQs5WsU0ZW82lDB8D3B6akJpYt7xAS5GI/EBBJhLGBtEKMHBt woU90dpPZtSrFg1B6+d8wX45yePcLUFsSZLrqSUNFks+4Gw6O8ObtjhumKSuvSNqSXwz QN6NQelFyddt/md5P76HIq3xxhU+aeFBL9ejrevxsx5VSM7X1yp8ntKPNRO6GJkVwpro qFCg== X-Gm-Message-State: AOAM530RVUrAT0OqSTrxwXxwVqI/ru7pv2O6pptAQBAPGCGyDIpKnp5d PRRzKU0iZtjfatWwpxlNCPwQiw== X-Google-Smtp-Source: ABdhPJzTZix3DrfI3gS99XuookHZB6B8CC1mG7Unokf53d9LadNonom0utrVmg95TnKUZqX4fQ7VFg== X-Received: by 2002:aa7:8f31:0:b029:1f8:987a:53dc with SMTP id y17-20020aa78f310000b02901f8987a53dcmr183958pfr.58.1615326193543; Tue, 09 Mar 2021 13:43:13 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id v3sm13672130pff.217.2021.03.09.13.43.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:11 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 6/7] arm64: entry: Enable random_kstack_offset support Date: Tue, 9 Mar 2021 13:43:00 -0800 Message-Id: <20210309214301.678739-7-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=ab928f13a74b1e09ed2d4740c77a8520d58ab562; i=6FjHrMJQ1a03WxuPiMt8aIK5MW/T0bayLG0SuUm7Wt4=; m=DW/OKOjAEidoxl6ODEV6zN0c7U/uCvK1t/uZsIkKPaM=; p=MOQyOeWlIC4XP1YdIGtrtyC9k7Ng+ZO0cCzQ0e3ME/8= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+QACgkQiXL039xtwCYAOg/+LgJ WgPJrpVY1ZhBup8fAF1iuohmC84qyEJf6H4fbXp63l60M3UxTJ/GNL23TdcEGxQ8swftFHFIkr6Nd xG8VuQNA3smCz7L5MqWyBsB2nf+SPSUYuC3mptJdGzraaD413rLhBOFojCAEDCWP7sYFRUtrV73am ZcSqH9n5FUVIbzACUtPaml/B7mEtFPB+lI6fJpLro72hGw6sjQJRkpiS5VhbWjJASeF47iChKcjBU l3m3qWTWSLt3SmZAaY+4XtHe1+TyOXw9RhR+erXEh6JmHpTQE6R4XRF9bkrRlUOboPGy502udXKds taG9ORm/JqzGUvKMUmgi1tjDPhPreYlYRNj4UaEIPLyPwfdbmatXCFyXM3YAkCKB74NE9rUI0W8pY w/dOS6d5U//pA346jb6D63ve1AMvz3boSI2IF0tm/BICqKURWY9oPV8h4wOM4FicJcBJNjRcwYK6x g0iwgAyle/C6bLnhgmWqApDv1NZ/boLi9Abl0qrQQxT7N1qfQOurTSsruSH0ZLauLKyQ7IGl6eELk HZ1Dxj2F1KBjPI2amq7hA7CrjogfXhsT5hqlw/f/By35OJk5gTSvOVGf5MY9LN4vd4o7O5FvboPlL XZvG0zwFnf+9JMrsXyGq0qjdoQuX9HtS1IQ7PM5EVEArilikYizdLC0wkQnHc5Cg= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214329_144062_0BED5C29 X-CRM114-Status: GOOD ( 16.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. (And include AAPCS rationale AAPCS thanks to Mark Rutland.) In order to avoid unconditional stack canaries on syscall entry (due to the use of alloca()), also disable stack protector to avoid triggering needless checks and slowing down the entry path. As there is no general way to control stack protector coverage with a function attribute[1], this must be disabled at the compilation unit level. This isn't a problem here, though, since stack protector was not triggered before: examining the resulting syscall.o, there are no changes in canary coverage (none before, none now). [1] a working __attribute__((no_stack_protector)) has been added to GCC and Clang but has not been released in any version yet: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=346b302d09c1e6db56d9fe69048acb32fbb97845 https://reviews.llvm.org/rG4fbf84c1732fca596ad1d6e96015e19760eb8a9b Signed-off-by: Kees Cook --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/Makefile | 5 +++++ arch/arm64/kernel/syscall.c | 10 ++++++++++ 3 files changed, 16 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1f212b47a48a..2d0e5f544429 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -146,6 +146,7 @@ config ARM64 select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT select HAVE_ARCH_PFN_VALID select HAVE_ARCH_PREL32_RELOCATIONS + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_STACKLEAK select HAVE_ARCH_THREAD_STRUCT_WHITELIST diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index ed65576ce710..6cc97730790e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -9,6 +9,11 @@ CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) +# Remove stack protector to avoid triggering unneeded stack canary +# checks due to randomize_kstack_offset. +CFLAGS_REMOVE_syscall.o = -fstack-protector -fstack-protector-strong +CFLAGS_syscall.o += -fno-stack-protector + # Object file lists. obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ entry-common.o entry-fpsimd.o process.o ptrace.o \ diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index b9cf12b271d7..58227a1c207e 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -43,6 +44,8 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, { long ret; + add_random_kstack_offset(); + if (scno < sc_nr) { syscall_fn_t syscall_fn; syscall_fn = syscall_table[array_index_nospec(scno, sc_nr)]; @@ -55,6 +58,13 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, ret = lower_32_bits(ret); regs->regs[0] = ret; + + /* + * The AAPCS mandates a 16-byte (i.e. 4-bit) aligned SP at + * function boundaries. We want at least 5 bits of entropy so we + * must randomize at least SP[8:4]. + */ + choose_random_kstack_offset(get_random_int() & 0x1FF); } static inline bool has_syscall_work(unsigned long flags) From patchwork Tue Mar 9 21:43:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12126615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86EBDC433DB for ; Tue, 9 Mar 2021 21:45:21 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2BEF464FD1 for ; Tue, 9 Mar 2021 21:45:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2BEF464FD1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/WanJRRO8OqtRhLSUCFjqjBWG5B+K2n3VxFvDWm09Vc=; b=i8lmqe+Fox6MfmdqGACo/lCt6 enNPM7ohvYSMA0HZOAZk47s45p9LIBuUwy7nJI1oKZ3JvLqRFXXBzOyH8A5xRa5IyTmIVYMF7HLvx HlZIf5AUdgABFL4pqdOihMpHnqQlfUpZR6PMFkd4ItsRocYb+RSVyt1cXwCp7dOuij8xrJSalOOVn jGLobdLk1AMuLkh5lsy3odYwmcIAViflpv4nnhi1igmq5Z9sZ7BW7iPKNizg1dlM6gGZaqwcbAxiI KHiYgyVQncNSFQzCtIMi8j4MQaraIC4pPlOdw8Z2sOhMHKPojhpVqw9/0KvcFrG7tN0V3TBXxG3uz 0CHj62u/Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8v-005U7k-Ao; Tue, 09 Mar 2021 21:43:53 +0000 Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lJk8M-005Txa-5J for linux-arm-kernel@lists.infradead.org; Tue, 09 Mar 2021 21:43:22 +0000 Received: by mail-pj1-x1031.google.com with SMTP id i14so1613801pjz.4 for ; Tue, 09 Mar 2021 13:43:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uk0/1LocoDticB91SF6FsgWpOKPsEr1Pb6N+E6L28KI=; b=fXCfvnU3D4wVS2gnx4R+6sEkANTA5C4Qx8s+R2Snvl4CfYrjd711MJmnMdXeC6GL/I PCoEcCoP0iSQ5ydymCVNPBPt0drSoZzLLX1BF0oLBizNtnX90NKPXVxlmNQlaA8fAIoj ryM7dBfe4KsBcn46bv7w6H7Idv2DnikOzytCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uk0/1LocoDticB91SF6FsgWpOKPsEr1Pb6N+E6L28KI=; b=rYzLbe8NqWjUsu5Aw9MO62hs5sxyy6d2mWp43qr85yCsQhrdh8pHFaJl53HkUP/gdj Seym/vUtBqQRP4Edb3V2UC5mPWMtAlDMpkcNbVF0BFkm5vFMDCkiNvQgIMCaJQlipqEN 8NOZEU7fIYH1dAbfvbU5s7NW1NPaDk8a8Y38ukYczHq8TNnsSIBhkBsycqA7/SGR2l4A S0DvjaOtsYPyQ0wQGa1Z9nfqEQKoQ4rmOXlRNBkP7Sg1v19gw/shWUZeLFNt2mmKg9sy wv+eDj5sFeMWtP0o6FTAy5/gvcKoNvIDwq7Xfyco60TZ5d9W7L23lX+5E/JlmPwmufM9 4ftQ== X-Gm-Message-State: AOAM530aXW66iJWiEyNDqOieDAwiZ32I3LnAUfBqewhXUOikQEBM99hQ vy++mZOoXJFg2bUpzZc2a6L59A== X-Google-Smtp-Source: ABdhPJwMAeJop9UPtI3jZOkzjoT12RbrdsqpFgJGTkl99HdMDta1V57qg5ZxtpfVe+IAGkz4ensuyA== X-Received: by 2002:a17:902:6ac1:b029:e3:dbc0:bc44 with SMTP id i1-20020a1709026ac1b02900e3dbc0bc44mr27861641plt.15.1615326193986; Tue, 09 Mar 2021 13:43:13 -0800 (PST) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id v3sm13672138pff.217.2021.03.09.13.43.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Mar 2021 13:43:11 -0800 (PST) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap Subject: [PATCH v5 7/7] lkdtm: Add REPORT_STACK for checking stack offsets Date: Tue, 9 Mar 2021 13:43:01 -0800 Message-Id: <20210309214301.678739-8-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210309214301.678739-1-keescook@chromium.org> References: <20210309214301.678739-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=fb815901a1ccc1d9c4ca5c3e3cd3729b7f382fe2; i=b69wRsxT78r/3tM1mGa7N6ME6+rlXyFg15giRWRwPAQ=; m=aFqgiEE+nAZdug79A1F+fVTg9ZceUb0WPE8cbHqssVg=; p=ZQ32/kILkW5AD3nBZHO0VMTp4prIPkm7+DdhCHX8KdA= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBH6+QACgkQiXL039xtwCY9qhAAhOW Jji3ZCJ3RnoaGRrD0g0EhW5yFisz4YxnlNFMM7jo9oEkkaEIg5Cms4k2RvZrJ99lKsWQwbBhzNuM+ rCDNkeT8iaL2KUsW7Wo7+6szheXR5qMdUTHKKX/nOd1uPS8zHdnMNBkOts94EJQ/Zi4WoE4M380XF 47N+kZ8mPuzp9vyHEVUuPre4Dc7Tqmg2bzpBi8tjTrZBVBXul/7Plp2Wbry/KX3lrUl/vq3VaNypY MhHZhQnufm5Nkx93kwAL/aB7zoIZppUx8G4GrqzgT8InUwgfWD/KaX6atmi0FiMIzyFRKfANnk/rG gm0DClhml0R0e5xMPFHy/aDTZxyjkarc66p/PSUfKCRWOfQ9mOA6HxIcpO8phsZ176kf1Ge+nrt0D vT+mGYXOznFNK8yNioU8/h3F6A/IUtFoK0BN1KTG5azbjperev/mix3MNVLHWQn64+OmoknnIZKKf boW/jcOs+FV/7/e11Atvz5H2BQyEer7KZmWnyoc+WZi//J69k8JN11FgMyMttnfU8h5qT3YzzT4F0 sEpmlrNFTfMHnL42Lq2rKO2uVjXqqzti1VNVAPVtpBgNCWdzhlBvF4VC+Y28uivtgwCovRhH4lKaO EcmyX88q25Tn95RBx7Y7EcFqRvubjTsJTcq3Dw9W08iqQrjt+/8Lnfz7XVUtGeJQ= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210309_214320_230461_8792487D X-CRM114-Status: GOOD ( 15.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For validating the stack offset behavior, report the offset from a given process's first seen stack address. A quick way to measure the entropy: for i in $(seq 1 1000); do echo "REPORT_STACK" >/sys/kernel/debug/provoke-crash/DIRECT done offsets=$(dmesg | grep 'Stack offset' | cut -d: -f3 | sort | uniq -c | sort -n | wc -l) echo "$(uname -m) bits of stack entropy: $(echo "obase=2; $offsets" | bc | wc -L)" Signed-off-by: Kees Cook --- drivers/misc/lkdtm/bugs.c | 17 +++++++++++++++++ drivers/misc/lkdtm/core.c | 1 + drivers/misc/lkdtm/lkdtm.h | 1 + 3 files changed, 19 insertions(+) diff --git a/drivers/misc/lkdtm/bugs.c b/drivers/misc/lkdtm/bugs.c index 110f5a8538e9..0e8254d0cf0b 100644 --- a/drivers/misc/lkdtm/bugs.c +++ b/drivers/misc/lkdtm/bugs.c @@ -134,6 +134,23 @@ noinline void lkdtm_CORRUPT_STACK_STRONG(void) __lkdtm_CORRUPT_STACK((void *)&data); } +static pid_t stack_pid; +static unsigned long stack_addr; + +void lkdtm_REPORT_STACK(void) +{ + volatile uintptr_t magic; + pid_t pid = task_pid_nr(current); + + if (pid != stack_pid) { + pr_info("Starting stack offset tracking for pid %d\n", pid); + stack_pid = pid; + stack_addr = (uintptr_t)&magic; + } + + pr_info("Stack offset: %d\n", (int)(stack_addr - (uintptr_t)&magic)); +} + void lkdtm_UNALIGNED_LOAD_STORE_WRITE(void) { static u8 data[5] __attribute__((aligned(4))) = {1, 2, 3, 4, 5}; diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c index b2aff4d87c01..8024b6a5cc7f 100644 --- a/drivers/misc/lkdtm/core.c +++ b/drivers/misc/lkdtm/core.c @@ -110,6 +110,7 @@ static const struct crashtype crashtypes[] = { CRASHTYPE(EXHAUST_STACK), CRASHTYPE(CORRUPT_STACK), CRASHTYPE(CORRUPT_STACK_STRONG), + CRASHTYPE(REPORT_STACK), CRASHTYPE(CORRUPT_LIST_ADD), CRASHTYPE(CORRUPT_LIST_DEL), CRASHTYPE(STACK_GUARD_PAGE_LEADING), diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h index 5ae48c64df24..99f90d3e5e9c 100644 --- a/drivers/misc/lkdtm/lkdtm.h +++ b/drivers/misc/lkdtm/lkdtm.h @@ -17,6 +17,7 @@ void lkdtm_LOOP(void); void lkdtm_EXHAUST_STACK(void); void lkdtm_CORRUPT_STACK(void); void lkdtm_CORRUPT_STACK_STRONG(void); +void lkdtm_REPORT_STACK(void); void lkdtm_UNALIGNED_LOAD_STORE_WRITE(void); void lkdtm_SOFTLOCKUP(void); void lkdtm_HARDLOCKUP(void);