From patchwork Mon Mar 15 18:02:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-22.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 269A5C433DB for ; Mon, 15 Mar 2021 18:04:49 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5017864F07 for ; Mon, 15 Mar 2021 18:04:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5017864F07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LkgS3CY9Q4oEb/64XiSFlIzSgogvN/68nhTe8LDmGgM=; b=qfgRx9qenAueSFM1gJrMKULFV DLRNne89ewFd+2KgQPX+wGhD0XsduTt/67ClERHjiuSkXxITmLcFVQWc4PaFKX8TIFQsd4GisAKOq mT4ET2LUxhUK2t0kuiCH0t/eo721VHdMRkhpYa8Tu5bQsXKeggN06C6buz/2WLRJcp429+et2FqA0 Q23hSthnsz1DZxaUicGTsCYyI2NVv7lC6R9ajoSI24l7ydLm1M/oQZjRTSd2hRuvv777nEfAcigv3 TJFuoHhdg+dHBwA+0ccCJS8qafN4tTqdT6YmyzLm4VZQ56xrHlGaolO+1kWE7xw81zi8T7kYy3P9R grgJO8GMg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLrYH-00Gcnc-2s; Mon, 15 Mar 2021 18:02:49 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrY5-00GchY-Oo for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:02:43 +0000 Received: by mail-pg1-x52d.google.com with SMTP id x29so20919780pgk.6 for ; Mon, 15 Mar 2021 11:02:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9ijlYcKqhmRnIjh3GfaZu6VzSGgGwccaBk76YZfh88U=; b=gBtfslh/2+D3O3puVX9ZDAtC36eVV5PwPxp4XA9eUO/GkNl99kAxZv1qBOO/2d87Yx sFcrGY7n+SrJGaLIEjGKQGsysYRo98xeJYmZG2aDXDED15Pp4X+bQ6GQU6WNAgGp/IQ/ RhvUNw3ubAilETtfi9v7RGV5r1E8XO5T7hEzk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9ijlYcKqhmRnIjh3GfaZu6VzSGgGwccaBk76YZfh88U=; b=mCXAb75LtcIjnCsi/PC8xqi4SXOn0r7QP/hu+e/AVcFWWksfqwkbR9NLRMTqxa//ro 6aaB8nj0E5OvuEAhV8I0SvLu/Iw/DCK0U7Hd2akQZbP0A7ntvRHhKj1psQLSVpdNfRKt gB8Tm9UBhzBugBEVbqord9xkhUqyuLH0L1jjJAC+l8AisjwwTNj+ygDYFeMQj5t10jRd WfrJx4/sb7hx8D7M1wVSgLyGDcrkA2OKc+zMVxrsiCw3glhfT4MiGgMtrtVZup/ZD1q1 NvQ6WjSpyNJOFA9g69sPZW7mBg9YmJJC5UDqs6NqT+6tMAmBicCgtyPX+Si24CBT3Zi2 AQaA== X-Gm-Message-State: AOAM533TSq140sqgjPQjo4cr4IZkZsSlPyk1exQhhg4Ta0g3DdiunjO/ ROa9zHG9YAkPZ9PUSdXnu9QI5Q== X-Google-Smtp-Source: ABdhPJzy+LhxO21xjzht2PAIEQOU5u9oqRHSnG2QWsS3Nj8lpin4Qjvz+HiWe9beyIvHlwAcX0i4Dg== X-Received: by 2002:a63:4848:: with SMTP id x8mr254310pgk.447.1615831356014; Mon, 15 Mar 2021 11:02:36 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id m6sm14381704pff.197.2021.03.15.11.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:35 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Peter Zijlstra , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 1/6] jump_label: Provide CONFIG-driven build state defaults Date: Mon, 15 Mar 2021 11:02:24 -0700 Message-Id: <20210315180229.1224655-2-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=aa54c44d1d71b9550d6015efc734f667917094a1; i=Vishx6UyAXwYzcnoSyP+eBB3iQyx+/i5smsbQfc0cnA=; m=vc4sSYlf+uaSlLSFP5TpbQv56VaSRpBLpRuMltXaB4Q=; p=mNH2Bo/K9vrGz9sBtTDV8UFO0eJ8yv8BbR/DeIaO1es= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTMACgkQiXL039xtwCbQphAAsvI HVl+t3wcEix5iAa5u+qee6jeL17WvN180P0khg9fTok8jiq8396y7TgIDHQ4BLkF581ql/0Uauyhr jQizhGDD1GLMceM5ndhY01fkg4Bes5IUnUBBbQH/gV8/7E43otmDhSiLdCOLcmlhx8pZIW1Opu7Mo EonqOy1s/+mgshZZgq4XXV3xTkI2qNLfCBEycbzwgtvwOQdrqAyYpOD3JSL0Vqliq7v4pvwC/ahth B6O1wLb8+MxSHDSyFokf2rXR+PXobQLKT2scaaXOWKOPcwaW6ZaRZMSNYgzK6xyZNGJFGyg8lr49Z yu9kYJSs1E1xWtkma0+bkxYQkq6n2dfL3v43ra5hW2Rl1zX7QCd4nWXr2YiI+41CEFaIFluRW5YrM vFvNr5CqfzZYj5Oj+y9vNImxUnuA80IpmpY7uxON7BldV817mlOGhWGeU5CaFGBW5IEpAeyp8zu8c 554NsrIbaSfKHKxQbKaMRoOjRWJtBIT5z4HMFnddOfv89W6uxZgYkwVHqkB2Y1cF6scB6BOAmL10+ fIha/mB1NPydLtzcs8vPg0ebIwYZrbixhhUOsMRc+BXlvvhXI2XVZ1YT3PpV21DYNs3o9xVsoVph6 PuU67dc/pi4YlA/CLOvmj3L2E4Khw2Kb6zsAoyE1S7cc8QQsVy+W5ZFEq4uOIfRg= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180238_225237_B2967BB9 X-CRM114-Status: GOOD ( 15.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As shown in jump_label.h[1], choosing the initial state of static branches changes the assembly layout. If the condition is expected to be likely it's inline, and if unlikely it is out of line via a jump. A few places in the kernel use (or could be using) a CONFIG to choose the default state, which would give a small performance benefit to their compile-time declared default. Provide the infrastructure to do this. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/linux/jump_label.h?h=v5.11#n398 Acked-by: Peter Zijlstra (Intel) Link: https://lore.kernel.org/lkml/20200324220641.GT2452@worktop.programming.kicks-ass.net/ Signed-off-by: Kees Cook --- include/linux/jump_label.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index d92691262f51..05f5554d860f 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -382,6 +382,21 @@ struct static_key_false { [0 ... (count) - 1] = STATIC_KEY_FALSE_INIT, \ } +#define _DEFINE_STATIC_KEY_1(name) DEFINE_STATIC_KEY_TRUE(name) +#define _DEFINE_STATIC_KEY_0(name) DEFINE_STATIC_KEY_FALSE(name) +#define DEFINE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_, IS_ENABLED(cfg))(name) + +#define _DEFINE_STATIC_KEY_RO_1(name) DEFINE_STATIC_KEY_TRUE_RO(name) +#define _DEFINE_STATIC_KEY_RO_0(name) DEFINE_STATIC_KEY_FALSE_RO(name) +#define DEFINE_STATIC_KEY_MAYBE_RO(cfg, name) \ + __PASTE(_DEFINE_STATIC_KEY_RO_, IS_ENABLED(cfg))(name) + +#define _DECLARE_STATIC_KEY_1(name) DECLARE_STATIC_KEY_TRUE(name) +#define _DECLARE_STATIC_KEY_0(name) DECLARE_STATIC_KEY_FALSE(name) +#define DECLARE_STATIC_KEY_MAYBE(cfg, name) \ + __PASTE(_DECLARE_STATIC_KEY_, IS_ENABLED(cfg))(name) + extern bool ____wrong_branch_error(void); #define static_key_enabled(x) \ @@ -482,6 +497,10 @@ extern bool ____wrong_branch_error(void); #endif /* CONFIG_JUMP_LABEL */ +#define static_branch_maybe(config, x) \ + (IS_ENABLED(config) ? static_branch_likely(x) \ + : static_branch_unlikely(x)) + /* * Advanced usage; refcount, branch is enabled when: count != 0 */ From patchwork Mon Mar 15 18:02:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60D1FC433DB for ; Mon, 15 Mar 2021 18:05:07 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9240C64F07 for ; Mon, 15 Mar 2021 18:05:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9240C64F07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gO68Qq0SfzFXJpuDxQ1g5n/xvBHDQQyLjahwZxoqAZQ=; b=cn39yyDNOabMxBxIQExSfbfZt 4T6FMN7eX8cCi6PYKvnFbCwdXn5hMEhCizij3XhBuGnTKzgCcvq0h2YM2sivYh8rR4zd8pyPGeP2d wwswrcRzVAMXrJrNlHOv2r01FdIrRCikU5LGxCLppS8n/b7NT+85+7ewvPKuxESa3XGpVFx4jAdSJ 0rTSPtMKzhTINvFWl3eCoQABIV9wjiQ34P08AGn397XX/ECPWAihiI/FaNG/bnGQ51e5c0V7n+XHo GTLxJieiZC7Sg06UqjZDnwQJ4haDewgslltN8UbuI1XsYVqLyjCvaCgTv7vDXQ5FWJ/dgUY60CVsP vSOXgqTZQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLrYe-00Gctb-W8; Mon, 15 Mar 2021 18:03:13 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrY5-00Gchf-Qn for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:02:43 +0000 Received: by mail-pl1-x636.google.com with SMTP id d23so12480436plq.2 for ; Mon, 15 Mar 2021 11:02:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K0uLi49Yx6Hs1kogqwqRCUz/Bpqh/dFOqLS4VRKyyBc=; b=NpVfbBSkCMGMjvqUBvpnwYBUDLSg0t4mYOsEdbBvi2yZSriP+lLClj248lVLlV1mpf nX/stjeHOgx3unJ6YY/jz4OctbXPuxLlEunQgCM/jxtNPHvbwL+KbU7nEMw6u8yj22PF /M7fJIqZgumlNjIVXD5qiAhMakD1OX3Kq+TYw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K0uLi49Yx6Hs1kogqwqRCUz/Bpqh/dFOqLS4VRKyyBc=; b=Neet3DJMhKDDddSDY5ihEt0HrABq8cQF7raxvngjfqP+ltyc1OPT7IhDmIatTbE0p6 Q7hg6eLPoQigQwsFC/UbNIDnbAhlPWOZUCFVaXtdj43Ta2KNTNtilTZ5280fiCTzu+Uu 1p/T/MmLSHSXMHBdCyDDgLy85rhJES2YbgsNztM+7dzfn9qHL3YmmxIl+f3gPpI6ppHy /4BwYJTaOKJGgO6SVTN962JLCztOqxF9L1os3a83Rsl/Nel34BBAFCpHVI0dtKEgWAqN C8EBNT4x4BP6wqc/wmtSq8ZUPeVG62RvX3tdXeTcOYa/ifZY4noh73V+c8Y75VxvEQfA nDyQ== X-Gm-Message-State: AOAM530fqo/S0le6OQXU9QFdR31TGk8KOtbEQrE2HVTM31q0CIsKmrEx UkFK/whg+gsIZz4cGWW3UgPJnQ== X-Google-Smtp-Source: ABdhPJw3XuSiNT39U8lLpLBLKNJyfyNfidC2LZuwiQfSz+u6PVqNYpahOsQIoKSVW+ozDdUAE8Umjg== X-Received: by 2002:a17:90b:691:: with SMTP id m17mr276901pjz.191.1615831356378; Mon, 15 Mar 2021 11:02:36 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id s194sm14416197pfs.57.2021.03.15.11.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:35 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Alexander Potapenko , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 2/6] init_on_alloc: Optimize static branches Date: Mon, 15 Mar 2021 11:02:25 -0700 Message-Id: <20210315180229.1224655-3-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=d73e14f50f59a92ac8538519f3e1f53d503ab26d; i=7VrM8Pzr5MGi8vAaDOW4xeiDtVYbjHgJsdLlT/sxDXE=; m=9pRPoTwktIPBeIWxjZ3xtu7JAekgWFD5JAmIfxnzZpU=; p=kGv6Q+SMrU5E1IdQ/vRaEbpDt4x68fXZGP3gRBxoslM= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTQACgkQiXL039xtwCYmkw//V/q 8YDIGvlrMf++Qpx3Y4F3jp6BV9YPPaIeh+7wAqmOGLpfhHGIggCuroF6Bx+Pb+rJbjJa63cbQizhi k+AxuhTsUnZpfHDR4Q1TIjsYJAN8XBXR1HGAaLwnq/D20E+EJsPOMDCisEsVPuvQbJdxaW7qb0oDK vmDWKOZCcN7VIYqpXnqB897l8P5RcLTtv4l3yUs1mVosqpb+CNXMSPI1Csr3e8nrZPM5SuoYZakj2 i3gfACSgTGgzqo5KZvY58pagDyHUK7lufPiP9kwsnmq6Jv8OvlbkNQwwC8vzmek5lJ9rpju01wlDr O4a7cCdjmm2CjDMAI1maK25DDY0MkNzViwrAtSJ0g28yVrPq8OTQINu4J/oKsu8DOeU8dk+34adFn 5fSkkgS3r20gMPIrsVPDewIJBmOzWQAUbaFhoUucmBcOAHLPnmmTyOzcBqBgksXuOckNOZcwIUvys Mtb04OUCS3Pxzl1b3XcOLInfxgqm88Bre7PgJv5HvexCmdLHKJ6aCNojW9UP4qngdA/HgFkL0LTwU PwcmH+LmUx5Ofpm8QT/VEoUhjsA3j9ZHX/R7vfwe9w6Sl42G6IXNZr9Ft79uOnYTfh3rkH8ksvs/U Xmq1KzeG9cpSvVSK3U4JH8tziH45YgiOwYUOazJKUTgYCIyV6QPse1xpMvnH/cXs= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180238_156595_5BA858B8 X-CRM114-Status: GOOD ( 17.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The state of CONFIG_INIT_ON_ALLOC_DEFAULT_ON (and ...ON_FREE...) did not change the assembly ordering of the static branches: they were always out of line. Use the new jump_label macros to check the CONFIG settings to default to the "expected" state, which slightly optimizes the resulting assembly code. Reviewed-by: Alexander Potapenko Link: https://lore.kernel.org/lkml/CAG_fn=X0DVwqLaHJTO6Jw7TGcMSm77GKHinrd0m_6y0SzWOrFA@mail.gmail.com/ Signed-off-by: Kees Cook Acked-by: Vlastimil Babka --- include/linux/mm.h | 10 ++++++---- mm/page_alloc.c | 4 ++-- mm/slab.h | 6 ++++-- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 77e64e3eac80..2ccd856ac0d1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2871,18 +2871,20 @@ static inline void kernel_poison_pages(struct page *page, int numpages) { } static inline void kernel_unpoison_pages(struct page *page, int numpages) { } #endif -DECLARE_STATIC_KEY_FALSE(init_on_alloc); +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc)) + if (static_branch_maybe(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, + &init_on_alloc)) return true; return flags & __GFP_ZERO; } -DECLARE_STATIC_KEY_FALSE(init_on_free); +DECLARE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free); + return static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free); } extern bool _debug_pagealloc_enabled_early; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3e4b29ee2b1e..267c04b8911d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -167,10 +167,10 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -DEFINE_STATIC_KEY_FALSE(init_on_alloc); +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc); EXPORT_SYMBOL(init_on_alloc); -DEFINE_STATIC_KEY_FALSE(init_on_free); +DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); static bool _init_on_alloc_enabled_early __read_mostly diff --git a/mm/slab.h b/mm/slab.h index 076582f58f68..b0977d525c06 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -601,7 +601,8 @@ static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) { - if (static_branch_unlikely(&init_on_alloc)) { + if (static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_alloc)) { if (c->ctor) return false; if (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON)) @@ -613,7 +614,8 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) static inline bool slab_want_init_on_free(struct kmem_cache *c) { - if (static_branch_unlikely(&init_on_free)) + if (static_branch_maybe(CONFIG_INIT_ON_FREE_DEFAULT_ON, + &init_on_free)) return !(c->ctor || (c->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_POISON))); return false; From patchwork Mon Mar 15 18:02:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48BF0C433E0 for ; Mon, 15 Mar 2021 18:11:08 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A373F64E28 for ; Mon, 15 Mar 2021 18:11:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A373F64E28 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cq6lSUnoJnchS5Ab/QSIaHtMErLFeWPuYWWvTIf5pdU=; b=glVxviOwjQszFjuhUCg5YK5gp gQBWlhHygs9HV0MX8bY87LWlRqi68uEuIix8Mpn+Q07JUFy1rnf6KR7cbJtqfDkte6tIvud04nYN2 yMJrGMsnuv6noYeLncM7UJQUUcF1U1am2CctfI+VVSoVXST/fNie0SGzTi18trYSjJNenn+v/4sMy zY3YCzHsWXZZFzu2yyKpjJuGiLCOkLfkI2lTigaIqnpCCia8YSATgnVfdb5Yse20hRvdDqmvVzYTs db1yHpcT+xrziT9BUi2IGNyauSjquwZ4NKXHIJRJcsfAyvTAT/jXJgZGacBcqoNI41OPQBG13y/SL 4brj81k/A==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLreu-00GeBN-Tk; Mon, 15 Mar 2021 18:09:41 +0000 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrel-00Ge8y-MM for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:09:38 +0000 Received: by mail-pj1-x1033.google.com with SMTP id lr10-20020a17090b4b8ab02900dd61b95c5eso12594697pjb.4 for ; Mon, 15 Mar 2021 11:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8RR/gKm83SIV1ashXcsFL/lRx7PJhODu8ZSMn6SRrJQ=; b=MmIc3nwGqROMT/7t71gH05s9mULUW5ZCpVenEvaSVse4ye3fbU1kGnVbMcpCylpe5g pGlX0maM33YQC0phbUx4s9s1YGXJGDmoBpuT8vrdx1unTdKMRKH3LcK1xH2V6wORaYvX 8LC3oRh4/EiArGYwkCB1iPg1pRCVgSwAprgyE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8RR/gKm83SIV1ashXcsFL/lRx7PJhODu8ZSMn6SRrJQ=; b=sY0UFWRXzP/PzviFqnTjKLECvkWED3Q/pxVDOpJSWk+5rSwrihMAXpur7nNdvWocQc OnR07RRNR7A9jDrB34Uy1OsXUP5gYVg8JN/2ixnDpGdVUctbKCewJUoaLc7LxIp2Pm8w 5v6XmyTWt0pu2QvSTXHkUk3NTQOltkVxDOck+udKDUu2vBxxA9H0LypyRyPe0CzJCTH2 5S4upYbRY2YpGvfFTuLD4WV3poox8B6nqUgzqHAkUIgbPxXKA6wKzE+LHSigk7mni9B5 tzEWi0X/OsEpWK3uHbGrUflu/Tj1fFHVlZnqp9ZGiq7zo90u9blmxSNx9e1Nm7kpXHdA 8m7g== X-Gm-Message-State: AOAM5336ZmMdDtnXOIghWH5waLEd8ifG56PKLZWcOJtLX58XfqfLM9St kaxXdoG4vBcK9SIa2hzIXIojm+dJwUHT9w== X-Google-Smtp-Source: ABdhPJzDUD/JdJg0PKMW3lr5YRQAEjHAT0vkNejlRAPlWFY8ENrWI4BWzRS668jmaxS5XfD68t7roQ== X-Received: by 2002:a17:902:ed91:b029:e6:125b:6bed with SMTP id e17-20020a170902ed91b02900e6125b6bedmr12890042plj.74.1615831358841; Mon, 15 Mar 2021 11:02:38 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id o9sm14914854pfh.47.2021.03.15.11.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:35 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 3/6] stack: Optionally randomize kernel stack offset each syscall Date: Mon, 15 Mar 2021 11:02:26 -0700 Message-Id: <20210315180229.1224655-4-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=4a65f719103e0800c3c67a72875df15eff769bc7; i=R0fcIKmEUh8zot7Dpk8JcJdIeXVfXkVsg5oQI7Df8iY=; m=MAP5vDRMDqR2Ss+UnPKNUGFW/8cuHmj3fGsgmrr5dWA=; p=FEZkCJfgrReOcdjubtlCDCLRXPUj5jJvx3leLrrKn8I= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTQACgkQiXL039xtwCZY0BAAjsd lGO68PQR3V5SFHF3n/fJJztIkqLXm0L0cRvdlpzEaODnm2IApuaOxPE7a+SUbnKiv9mHpBK9hkvB9 Bfw6KHYfYyhpHg43qbpcnXRgRnxIPNrI4ciHYNDcs7tEL09kXJhEh9mER8jtCk8b3ml7uNug9YzPR nuSwufLkckNvzsZl8LBMKlJYDPA1Us/s9NyKJ8mVQnPeFTF7LZJipZH+oZ5CGRFaw6sN0qpD2GReX OJgkmo2zca8KO9ByZFykXrii7z+6/WDHv1dloDAbPOvDHvRCngKZhp4Wwc4JJwUtaDo4xQ7+461tT x+6w6oGRKapXCrSrUHITiYHqCa0bkCQO8X2mNaHx010ULLqDm/eOPk0hMkdKsGE2TMxTDJER692oj N4MRsmEvluWAG1OUYTa8P87cxjH5Pq1tn3swP3SHrMxw3pCgNASZuFWCS1DNL1P7Jkq3cblMoLnh8 0N0Q1GQogAtlCk4DvSvANOxZViCOEk7d+dQHo1gR0qiu+MZcV6AnJD2C9WpLyEC7b2z1Q38P2h3FL vHtdMwXRHsIK0EJJp0SB/HevRbNHweOzO7hv0M9s0lQsDGjIGF1LtRqQgC6sGd2bb6nLweW9GbTr9 E7T9qbNUbvwdeIQaNEW2FY88Qh5J5p7w35YeW8r3c4iJdDGnppUJdKVayXoABBF8= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180932_235293_A4938EAD X-CRM114-Status: GOOD ( 38.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This provides the ability for architectures to enable kernel stack base address offset randomization. This feature is controlled by the boot param "randomize_kstack_offset=on/off", with its default value set by CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. This feature is based on the original idea from the last public release of PaX's RANDKSTACK feature: https://pax.grsecurity.net/docs/randkstack.txt All the credit for the original idea goes to the PaX team. Note that the design and implementation of this upstream randomize_kstack_offset feature differs greatly from the RANDKSTACK feature (see below). Reasoning for the feature: This feature aims to make harder the various stack-based attacks that rely on deterministic stack structure. We have had many such attacks in past (just to name few): https://jon.oberheide.org/files/infiltrate12-thestackisback.pdf https://jon.oberheide.org/files/stackjacking-infiltrate11.pdf https://googleprojectzero.blogspot.com/2016/06/exploiting-recursion-in-linux-kernel_20.html As Linux kernel stack protections have been constantly improving (vmap-based stack allocation with guard pages, removal of thread_info, STACKLEAK), attackers have had to find new ways for their exploits to work. They have done so, continuing to rely on the kernel's stack determinism, in situations where VMAP_STACK and THREAD_INFO_IN_TASK_STRUCT were not relevant. For example, the following recent attacks would have been hampered if the stack offset was non-deterministic between syscalls: https://repositorio-aberto.up.pt/bitstream/10216/125357/2/374717.pdf (page 70: targeting the pt_regs copy with linear stack overflow) https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html (leaked stack address from one syscall as a target during next syscall) The main idea is that since the stack offset is randomized on each system call, it is harder for an attack to reliably land in any particular place on the thread stack, even with address exposures, as the stack base will change on the next syscall. Also, since randomization is performed after placing pt_regs, the ptrace-based approach[1] to discover the randomized offset during a long-running syscall should not be possible. Design description: During most of the kernel's execution, it runs on the "thread stack", which is pretty deterministic in its structure: it is fixed in size, and on every entry from userspace to kernel on a syscall the thread stack starts construction from an address fetched from the per-cpu cpu_current_top_of_stack variable. The first element to be pushed to the thread stack is the pt_regs struct that stores all required CPU registers and syscall parameters. Finally the specific syscall function is called, with the stack being used as the kernel executes the resulting request. The goal of randomize_kstack_offset feature is to add a random offset after the pt_regs has been pushed to the stack and before the rest of the thread stack is used during the syscall processing, and to change it every time a process issues a syscall. The source of randomness is currently architecture-defined (but x86 is using the low byte of rdtsc()). Future improvements for different entropy sources is possible, but out of scope for this patch. As suggested by Andy Lutomirski, the offset is added using alloca() and an empty asm() statement with an output constraint, since it avoid changes to assembly syscall entry code, to the unwinder, and provides correct stack alignment as defined by the compiler. In order to make this available by default with zero performance impact for those that don't want it, it is boot-time selectable with static branches. This way, if the overhead is not wanted, it can just be left turned off with no performance impact. The generated assembly for x86_64 with GCC looks like this: ... ffffffff81003977: 65 8b 05 02 ea 00 7f mov %gs:0x7f00ea02(%rip),%eax # 12380 ffffffff8100397e: 25 ff 03 00 00 and $0x3ff,%eax ffffffff81003983: 48 83 c0 0f add $0xf,%rax ffffffff81003987: 25 f8 07 00 00 and $0x7f8,%eax ffffffff8100398c: 48 29 c4 sub %rax,%rsp ffffffff8100398f: 48 8d 44 24 0f lea 0xf(%rsp),%rax ffffffff81003994: 48 83 e0 f0 and $0xfffffffffffffff0,%rax ... As a result of the above stack alignment, this patch introduces about 5 bits of randomness after pt_regs is spilled to the thread stack on x86_64, and 6 bits on x86_32 (since its has 1 fewer bit required for stack alignment). The amount of entropy could be adjusted based on how much of the stack space we wish to trade for security. My measure of syscall performance overhead (on x86_64): lmbench: /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_syscall -N 10000 null randomize_kstack_offset=y Simple syscall: 0.7082 microseconds randomize_kstack_offset=n Simple syscall: 0.7016 microseconds So, roughly 0.9% overhead growth for a no-op syscall, which is very manageable. And for people that don't want this, it's off by default. There are two gotchas with using the alloca() trick. First, compilers that have Stack Clash protection (-fstack-clash-protection) enabled by default (e.g. Ubuntu[3]) add pagesize stack probes to any dynamic stack allocations. While the randomization offset is always less than a page, the resulting assembly would still contain (unreachable!) probing routines, bloating the resulting assembly. To avoid this, -fno-stack-clash-protection is unconditionally added to the kernel Makefile since this is the only dynamic stack allocation in the kernel (now that VLAs have been removed) and it is provably safe from Stack Clash style attacks. The second gotcha with alloca() is a negative interaction with -fstack-protector*, in that it sees the alloca() as an array allocation, which triggers the unconditional addition of the stack canary function pre/post-amble which slows down syscalls regardless of the static branch. In order to avoid adding this unneeded check and its associated performance impact, architectures need to carefully remove uses of -fstack-protector-strong (or -fstack-protector) in the compilation units that use the add_random_kstack() macro and to audit the resulting stack mitigation coverage (to make sure no desired coverage disappears). No change is visible for this on x86 because the stack protector is already unconditionally disabled for the compilation unit, but the change is required on arm64. There is, unfortunately, no attribute that can be used to disable stack protector for specific functions. Comparison to PaX RANDKSTACK feature: The RANDKSTACK feature randomizes the location of the stack start (cpu_current_top_of_stack), i.e. including the location of pt_regs structure itself on the stack. Initially this patch followed the same approach, but during the recent discussions[2], it has been determined to be of a little value since, if ptrace functionality is available for an attacker, they can use PTRACE_PEEKUSR/PTRACE_POKEUSR to read/write different offsets in the pt_regs struct, observe the cache behavior of the pt_regs accesses, and figure out the random stack offset. Another difference is that the random offset is stored in a per-cpu variable, rather than having it be per-thread. As a result, these implementations differ a fair bit in their implementation details and results, though obviously the intent is similar. [1] https://lore.kernel.org/kernel-hardening/2236FBA76BA1254E88B949DDB74E612BA4BC57C1@IRSMSX102.ger.corp.intel.com/ [2] https://lore.kernel.org/kernel-hardening/20190329081358.30497-1-elena.reshetova@intel.com/ [3] https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040741.html Co-developed-by: Elena Reshetova Signed-off-by: Elena Reshetova Link: https://lore.kernel.org/r/20190415060918.3766-1-elena.reshetova@intel.com Signed-off-by: Kees Cook --- .../admin-guide/kernel-parameters.txt | 11 +++++ Makefile | 4 ++ arch/Kconfig | 23 ++++++++++ include/linux/randomize_kstack.h | 42 +++++++++++++++++++ init/main.c | 23 ++++++++++ 5 files changed, 103 insertions(+) create mode 100644 include/linux/randomize_kstack.h diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 04545725f187..bee8644a192e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4061,6 +4061,17 @@ fully seed the kernel's CRNG. Default is controlled by CONFIG_RANDOM_TRUST_CPU. + randomize_kstack_offset= + [KNL] Enable or disable kernel stack offset + randomization, which provides roughly 5 bits of + entropy, frustrating memory corruption attacks + that depend on stack address determinism or + cross-syscall address exposures. This is only + available on architectures that have defined + CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET. + Format: (1/Y/y=enable, 0/N/n=disable) + Default is CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT. + ras=option[,option,...] [KNL] RAS-specific options cec_disable [X86] diff --git a/Makefile b/Makefile index 31dcdb3d61fa..8a959a264588 100644 --- a/Makefile +++ b/Makefile @@ -811,6 +811,10 @@ KBUILD_CFLAGS += -ftrivial-auto-var-init=zero KBUILD_CFLAGS += -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang endif +# While VLAs have been removed, GCC produces unreachable stack probes +# for the randomize_kstack_offset feature. Disable it for all compilers. +KBUILD_CFLAGS += $(call cc-option, -fno-stack-clash-protection) + DEBUG_CFLAGS := # Workaround for GCC versions < 5.0 diff --git a/arch/Kconfig b/arch/Kconfig index 2bb30673d8e6..4fe6b047fcbc 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1055,6 +1055,29 @@ config VMAP_STACK backing virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. +config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + def_bool n + help + An arch should select this symbol if it can support kernel stack + offset randomization with calls to add_random_kstack_offset() + during syscall entry and choose_random_kstack_offset() during + syscall exit. Careful removal of -fstack-protector-strong and + -fstack-protector should also be applied to the entry code and + closely examined, as the artificial stack bump looks like an array + to the compiler, so it will attempt to add canary checks regardless + of the static branch state. + +config RANDOMIZE_KSTACK_OFFSET_DEFAULT + bool "Randomize kernel stack offset on syscall entry" + depends on HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET + help + The kernel stack offset can be randomized (after pt_regs) by + roughly 5 bits of entropy, frustrating memory corruption + attacks that depend on stack address determinism or + cross-syscall address exposures. This feature is controlled + by kernel boot param "randomize_kstack_offset=on/off", and this + config chooses the default boot state. + config ARCH_OPTIONAL_KERNEL_RWX def_bool n diff --git a/include/linux/randomize_kstack.h b/include/linux/randomize_kstack.h new file mode 100644 index 000000000000..c4701a39c21f --- /dev/null +++ b/include/linux/randomize_kstack.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _LINUX_RANDOMIZE_KSTACK_H +#define _LINUX_RANDOMIZE_KSTACK_H + +#include +#include +#include + +DECLARE_STATIC_KEY_MAYBE(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DECLARE_PER_CPU(u32, kstack_offset); + +/* + * Do not use this anywhere else in the kernel. This is used here because + * it provides an arch-agnostic way to grow the stack with correct + * alignment. Also, since this use is being explicitly masked to a max of + * 10 bits, stack-clash style attacks are unlikely. For more details see + * "VLAs" in Documentation/process/deprecated.rst + * The asm statement is designed to convince the compiler to keep the + * allocation around even after "ptr" goes out of scope. + */ +void *__builtin_alloca(size_t size); + +#define add_random_kstack_offset() do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + u8 *ptr = __builtin_alloca(offset & 0x3FF); \ + asm volatile("" : "=m"(*ptr) :: "memory"); \ + } \ +} while (0) + +#define choose_random_kstack_offset(rand) do { \ + if (static_branch_maybe(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, \ + &randomize_kstack_offset)) { \ + u32 offset = this_cpu_read(kstack_offset); \ + offset ^= (rand); \ + this_cpu_write(kstack_offset, offset); \ + } \ +} while (0) + +#endif diff --git a/init/main.c b/init/main.c index 53b278845b88..f498aac26e8c 100644 --- a/init/main.c +++ b/init/main.c @@ -844,6 +844,29 @@ static void __init mm_init(void) pti_init(); } +#ifdef CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET +DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, + randomize_kstack_offset); +DEFINE_PER_CPU(u32, kstack_offset); + +static int __init early_randomize_kstack_offset(char *buf) +{ + int ret; + bool bool_result; + + ret = kstrtobool(buf, &bool_result); + if (ret) + return ret; + + if (bool_result) + static_branch_enable(&randomize_kstack_offset); + else + static_branch_disable(&randomize_kstack_offset); + return 0; +} +early_param("randomize_kstack_offset", early_randomize_kstack_offset); +#endif + void __init __weak arch_call_rest_init(void) { rest_init(); From patchwork Mon Mar 15 18:02:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2896EC433DB for ; Mon, 15 Mar 2021 18:05:46 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B9B6164F2A for ; Mon, 15 Mar 2021 18:05:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9B6164F2A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RyROqPeD3DMGvyCWn1m9b0vfAX88EY6e81zqEbAncfg=; b=fmNQNLs8Smt0QiYikl3lolbFL KvX1kcE4CoEkNWGpOewdIH827KIiMlRJtfkU66xNa8BTUggIXD5Fsj82Y5zx6jEscepxToznMzGGM Bg9L5aH+d100LqF/Fr517aJtKn2uv47rzpf/aSojentAK9VVn5LLIfC5T9NA5u8ol7hztaHppSG7X /oZQmZGMXD6El/fZ5KRIIappwKaFSotnblwjXhf3XPsm1L3YRbBEKYd5tSUZBPvJEKSANiJGLX6uI eUPyV3fdjbb6pJCzM6lu3f6aqi9oGfp0TYlVDmcjxWaMbiKWDs2X0ByN0iRyIXNefDqsZ0fVGm9kn qbUzhm6Vg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLrZQ-00Gd2r-2F; Mon, 15 Mar 2021 18:04:00 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrY8-00GckL-8K for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:02:48 +0000 Received: by mail-pf1-x436.google.com with SMTP id c17so3496123pfv.12 for ; Mon, 15 Mar 2021 11:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DcTnVacz4A0tmKEIIvO2n/EpgXlfdRl1ckINTgAR5oI=; b=PQ3XGnvMsKcXU4s5p+F5dfZF6FA/iSbcSn0nswwcqPExK09KIRLEltkz0Bh8wiwE7N 1XkNp1GfB55SBD6zDGJQRjdRpTR4DYjid5msGEofOSSYqbipdCIbWhBqrPM9AWy1UM52 l+EaNe1XtG5skHL76gK0sXV38Rb8Lgh4egJtA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DcTnVacz4A0tmKEIIvO2n/EpgXlfdRl1ckINTgAR5oI=; b=Cx0s09rrSIJCriO7J8MQNaopukqrpgRKvA45WL1Fr6W1hvn4tSbCmiCizaGXjPn9SQ 50Nhc3LIQzpZJ3sTUUf58wt3KCXpHb1J+LnE5Bhq+4jAKi4tWT+nNZzVohKwxuwqpRgk M/PUji0+lxjwCrsXVlPRozV5DwTStYgYuzL1O5DjqutRfxnbtZ9CYdQtWOG7HXpSIPvy lDXLEFUiPbokfUMWY9TYuHdQGrTUjXGKaals4xVga4l+z9mlZru4ZmFQjjZTp/B/5/UX 8AbIQ6rODZZ5sO1DhM2LgQsIgVn5Bxs35+nnEAkT6yPMVWC07gSFr9QK62BxYA9VSagH uZeQ== X-Gm-Message-State: AOAM532Jq3TN6E8mXXHCgpsoTN38wcdy3oO3kR7EDG9TX/9YrAciVrnN 3VjWQxWZocGfwy0uG0Aja/Jkzw== X-Google-Smtp-Source: ABdhPJxy/jBx4lbYUG9dTL/BSOwG++1sLNIRxLhlf/Aak5xk+P2rM72P6uwME362sOaUIYrG0OIg4w== X-Received: by 2002:a65:4c86:: with SMTP id m6mr314819pgt.174.1615831358475; Mon, 15 Mar 2021 11:02:38 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id e20sm13764419pgm.1.2021.03.15.11.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:35 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 4/6] x86/entry: Enable random_kstack_offset support Date: Mon, 15 Mar 2021 11:02:27 -0700 Message-Id: <20210315180229.1224655-5-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=1a3993b6371bcc7c9fa1576e563a724a70c25875; i=np7yed3mY+gWIWkexmB7CyDLgwsIh0xV2RGaksJc7tI=; m=cc9SLz8pm+mdXOKZLHtRjH7FITDzwHCOKmwFrX1p7io=; p=UyRR1nzRrbUXeUU5ICay1xt2wYZVgu7b/Eh0FjkN2Hc= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTQACgkQiXL039xtwCb9dw/8CDN Ir13NcCLMJOuRm2LpzDYbdgOc9JQ2QokpXPH/jLGBVXWXwV2g8Nk5TYJJcm7X/yj75HbSyNJLBUVU c8QbEGqzcBxjyIZUTKBDrYfApusLejywRj5YCmq1CONO5XBcJTMPuNAo1ZlzyQ0fI+754FraF3eU/ O8qcOD18DBVIsRP7e9ASpSyjqINb46mpEDrlvXa+uJyzGfAGLz+B0NLm6WPK0eaKzRmPC3mUs5gXU 9pL8/JlmcsD3Ne3kzSexmn7+uvYXEcwRagE8A76gkzE69JK3Lq8uIAyFARV2VO8mXZcUD/loxowRK vpn44j3EK04VB/Nf0IEDP6SNkNmSFvdqOSbp8gXcBI+rAJgA91SXLBf+ld9pIsIkGf6+uvY+rKqb7 XT7pyiu/3f7C3m64R7FFFUsWH6waSp2VR4ccIbNM97N6o5FY5keipPsjJNinKedZwOGjmY2O+FMkE nxUr2FjxcwFOBCbAkwJL+rgVv9UPOprPnq+Gxi6axTrDjRe3JoWtapLhbdDjHYLos7Gh5+UhghIj/ vYKfMHbHOOlKyhcrufiMXbcy0dkK6EK9ES3KNDmWu0cpxS2QAPrJvyrC6eCIxT9T0fko/eWgdUhWi ZNc8RzQeH2jVopqqCkmd8L74weZxz2gnh8S5w2OW3/yb5VWZ8VjJWDr3879m6aVE= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180240_660136_2BA0A232 X-CRM114-Status: GOOD ( 18.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5-6 bits of entropy, depending on compiler and word size. Since the method of offsetting uses macros, this cannot live in the common entry code (the stack offset needs to be retained for the life of the syscall, which means it needs to happen at the actual entry point). Signed-off-by: Kees Cook --- arch/x86/Kconfig | 1 + arch/x86/entry/common.c | 3 +++ arch/x86/include/asm/entry-common.h | 8 ++++++++ 3 files changed, 12 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2792879d398e..4b4ad8ec10d2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -165,6 +165,7 @@ config X86 select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if X86_64 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index a2433ae8a65e..810983d7c26f 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -38,6 +38,7 @@ #ifdef CONFIG_X86_64 __visible noinstr void do_syscall_64(unsigned long nr, struct pt_regs *regs) { + add_random_kstack_offset(); nr = syscall_enter_from_user_mode(regs, nr); instrumentation_begin(); @@ -83,6 +84,7 @@ __visible noinstr void do_int80_syscall_32(struct pt_regs *regs) { unsigned int nr = syscall_32_enter(regs); + add_random_kstack_offset(); /* * Subtlety here: if ptrace pokes something larger than 2^32-1 into * orig_ax, the unsigned int return value truncates it. This may @@ -102,6 +104,7 @@ static noinstr bool __do_fast_syscall_32(struct pt_regs *regs) unsigned int nr = syscall_32_enter(regs); int res; + add_random_kstack_offset(); /* * This cannot use syscall_enter_from_user_mode() as it has to * fetch EBP before invoking any of the syscall entry work diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index 2b87b191b3b8..8e41566e154a 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -2,6 +2,7 @@ #ifndef _ASM_X86_ENTRY_COMMON_H #define _ASM_X86_ENTRY_COMMON_H +#include #include #include @@ -70,6 +71,13 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, */ current_thread_info()->status &= ~(TS_COMPAT | TS_I386_REGS_POKED); #endif + + /* + * x86_64 stack alignment means 3 bits are ignored, so keep + * the top 5 bits. x86_32 needs only 2 bits of alignment, so + * the top 6 bits will be used. + */ + choose_random_kstack_offset(rdtsc() & 0xFF); } #define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare From patchwork Mon Mar 15 18:02:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A0A6C433DB for ; Mon, 15 Mar 2021 18:11:09 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DCEA064F48 for ; Mon, 15 Mar 2021 18:11:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DCEA064F48 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=S0tApvcGz0BPZalyNfyrQot0aZqj4AEzcL4QokvdNhw=; b=gvgyeZueqlxvQA+CqC/cXwB8P 8VSMSCGF89JZ/WXdwunay3VzmuKsJv/rHBTR45EQCqhZafzGlVz9LlFJOakyAKp0RQkvIMntlhgtB yv2SMDY5RbNIUevMoir2oYGsI/7yusYiS2ipRs0U2ZFzMqyMSo7PC64/2x5f9iq5rxUe1xLEa8lQY VYruz/L+o6pzytNh4TklVEaZKz2qkVRhTAwE6+ukZo31XM8f6a/yyZxAZWBjyXwDhuSOX8nJld7xU DyshrkIhfZZeBZaOwDB6yHP/AqiFoqk2bBJx3Wtj67OtdIC48VoUPsrJLJvsBCG+rxpPi2iUVqviY KYmbQYA3Q==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLrf8-00GeFH-T8; Mon, 15 Mar 2021 18:09:55 +0000 Received: from mail-io1-xd30.google.com ([2607:f8b0:4864:20::d30]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrex-00GeBs-Ff for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:09:46 +0000 Received: by mail-io1-xd30.google.com with SMTP id u20so34353845iot.9 for ; Mon, 15 Mar 2021 11:09:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yYr9wVtQr+Yv2YCxxAWFJhlutYOKWxR3zpP8Meqzwc0=; b=PJ3T01c5ZmSw41q5MAxkemIUNEX541whNOtK/oVGkgDvEXke5s57A/1q9tzDzo+AEJ 17TwxsQhxMpKr12YLaf51slHhYzShk72wq3LA7Tf1jp2ZGhrKU21lUBPzS/02MQjUN4a 8BNS0yFQ9mNrfIeZXkahph53aQpbFJsZAfitI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yYr9wVtQr+Yv2YCxxAWFJhlutYOKWxR3zpP8Meqzwc0=; b=QzAGVrSMwEdg228Y4AqIbQhZY8YlgISPIBMZhLL8JWZKNmmezrisYWj13TP8BDt/O0 ET0qk9Nt87AnFcdDKzqTTPNAbhm7rJJK8oAl/1tFENZAOwzaddBo+GD32VL9dQas82dS TBn+mEiUiKY70T88SdwB2aKrfnIJV7ca0POA+q/QYZFfkmhdYEzTn6ErEj2qg9M/QYzC SLL+mvcXTqSDvMgnUqMzIA42GZFsucZMJgLn5u8AYP4dGL7KArTcxTuT6xcygxAN8eG2 sI6bwPbgRbwyFt4ouDRciNrdvCNDj1Lp6ptaCeSKe3dyRh2wrCw64bKWD4JiQWeBM4FT sniQ== X-Gm-Message-State: AOAM530O2Jf57d5dHf8I2FVXGCotvGipeQ7EaLh1VBk9LALqS0qRjSBi wdNfoSP16yMDCkRb6gup+dk4C9Wgp+YQHQ== X-Google-Smtp-Source: ABdhPJypLz3rt4xSDEfwAQI69keGjt8wki/cYUqoCZg01pG+rs+SU3iIC2Rhxqq4jRLey7oHOmrHHw== X-Received: by 2002:a65:6208:: with SMTP id d8mr260022pgv.365.1615831359582; Mon, 15 Mar 2021 11:02:39 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id w188sm14531942pfw.177.2021.03.15.11.02.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:38 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 5/6] arm64: entry: Enable random_kstack_offset support Date: Mon, 15 Mar 2021 11:02:28 -0700 Message-Id: <20210315180229.1224655-6-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=ab928f13a74b1e09ed2d4740c77a8520d58ab562; i=6FjHrMJQ1a03WxuPiMt8aIK5MW/T0bayLG0SuUm7Wt4=; m=DW/OKOjAEidoxl6ODEV6zN0c7U/uCvK1t/uZsIkKPaM=; p=MOQyOeWlIC4XP1YdIGtrtyC9k7Ng+ZO0cCzQ0e3ME/8= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTQACgkQiXL039xtwCZ8+hAAok1 EQH02pRdoiRp9m8tbYTx8Kv8urKXluNaNLKQ8VjvRmjYSBNnvpTWI/n09jSnhzsXmY3lP9KFBiiSR vTF18ZmisLjVh1++NAWmL4zJNQExvXHxdxXvrzs6I0wIm+3ULPb+9DxBwwzeFIOlTC0AHfkarOIvI JPzaUMXvv+p+TiFW12lULVfRIkjZeTY8diwqSzXhh0Z+sJFZt0H5C9CzgjNlKMnNDGj2r4ie5i14O 4pSTw5xa/wRNzIDemET+Cl6r5cxlzMmyUnwKxL8cQl2HF2ysqiZFcCqUrsFdHWMiO8p9js8+hJjdr DnVs/0zGx9OvIObQq8Cs6RXEWpOQT91MbY6iWdLYoP1I+4vwxcR7P2bIm8kvDtL/P+YEVghQCMAi/ Nbw7LFXshsWv4OkJT8XZvnuYTdN/6HIL6Pz1V74YC5b4+90rxYBfG6Di9MIxsJk6X3QQh6+yznoDJ YmtysrTNSPGMXjKDJfs3f3yKIpWSePYSb9w6LZMFUww5o0vdipqLGAv4onWHWsatjKjTN7Qi8Qy2v z/gu45senClYLMA4IqByUxpO7K0pVhkjIJ9bCuGqylQg37zdDD/h9fh1xGKCCzDeGBHjt88zBIQy0 1RmtN5iz/6GO6LViI+zmCV5RpDsDtBtZA7ZIuS5jlamNSIEmDoQBVdxK0QuCHdbw= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180943_624514_AF16B1A4 X-CRM114-Status: GOOD ( 18.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow for a randomized stack offset on a per-syscall basis, with roughly 5 bits of entropy. (And include AAPCS rationale AAPCS thanks to Mark Rutland.) In order to avoid unconditional stack canaries on syscall entry (due to the use of alloca()), also disable stack protector to avoid triggering needless checks and slowing down the entry path. As there is no general way to control stack protector coverage with a function attribute[1], this must be disabled at the compilation unit level. This isn't a problem here, though, since stack protector was not triggered before: examining the resulting syscall.o, there are no changes in canary coverage (none before, none now). [1] a working __attribute__((no_stack_protector)) has been added to GCC and Clang but has not been released in any version yet: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;h=346b302d09c1e6db56d9fe69048acb32fbb97845 https://reviews.llvm.org/rG4fbf84c1732fca596ad1d6e96015e19760eb8a9b Signed-off-by: Kees Cook --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/Makefile | 5 +++++ arch/arm64/kernel/syscall.c | 10 ++++++++++ 3 files changed, 16 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1f212b47a48a..2d0e5f544429 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -146,6 +146,7 @@ config ARM64 select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT select HAVE_ARCH_PFN_VALID select HAVE_ARCH_PREL32_RELOCATIONS + select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_STACKLEAK select HAVE_ARCH_THREAD_STRUCT_WHITELIST diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index ed65576ce710..6cc97730790e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -9,6 +9,11 @@ CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_insn.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_return_address.o = $(CC_FLAGS_FTRACE) +# Remove stack protector to avoid triggering unneeded stack canary +# checks due to randomize_kstack_offset. +CFLAGS_REMOVE_syscall.o = -fstack-protector -fstack-protector-strong +CFLAGS_syscall.o += -fno-stack-protector + # Object file lists. obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ entry-common.o entry-fpsimd.o process.o ptrace.o \ diff --git a/arch/arm64/kernel/syscall.c b/arch/arm64/kernel/syscall.c index b9cf12b271d7..58227a1c207e 100644 --- a/arch/arm64/kernel/syscall.c +++ b/arch/arm64/kernel/syscall.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -43,6 +44,8 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, { long ret; + add_random_kstack_offset(); + if (scno < sc_nr) { syscall_fn_t syscall_fn; syscall_fn = syscall_table[array_index_nospec(scno, sc_nr)]; @@ -55,6 +58,13 @@ static void invoke_syscall(struct pt_regs *regs, unsigned int scno, ret = lower_32_bits(ret); regs->regs[0] = ret; + + /* + * The AAPCS mandates a 16-byte (i.e. 4-bit) aligned SP at + * function boundaries. We want at least 5 bits of entropy so we + * must randomize at least SP[8:4]. + */ + choose_random_kstack_offset(get_random_int() & 0x1FF); } static inline bool has_syscall_work(unsigned long flags) From patchwork Mon Mar 15 18:02:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 12140333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69AD6C433DB for ; Mon, 15 Mar 2021 18:09:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F206364F48 for ; Mon, 15 Mar 2021 18:09:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F206364F48 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/WanJRRO8OqtRhLSUCFjqjBWG5B+K2n3VxFvDWm09Vc=; b=Kmnjk4L9Fn8qW5Nglcm9mnN2V 6xNe1mESJrB24y1l6zhT51IlEIehCtx64o3z3fwzNGcUvQ+YNfkLMrKcWJ1ixaIeUsLUiVyyo8hZb LEUIhK0pYOmJFgLaJQogBIDzzJ0nRoFo/2MiBq9UCOlxocp3SGIs8FMrJAcSZJz9+4TOmhA/aA9iJ pwy+A+JbIqGP6ks7syE/G37wg0Im4hjmAp0xTkw2R7VESuRiiyCixhllySuXpLMznhZGUrMBfABSh NCZ07p7joP712qWC3QY2i7WNtx+007wIhkuLAVgCZl7uja/lgo0BjB5tJ7cK6UL0uQoxasrlj/EIw KYPb8iUPg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lLrdS-00Gdu8-3B; Mon, 15 Mar 2021 18:08:10 +0000 Received: from mail-qv1-xf2d.google.com ([2607:f8b0:4864:20::f2d]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lLrdN-00Gdt4-DT for linux-arm-kernel@lists.infradead.org; Mon, 15 Mar 2021 18:08:07 +0000 Received: by mail-qv1-xf2d.google.com with SMTP id j17so8224724qvo.13 for ; Mon, 15 Mar 2021 11:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uk0/1LocoDticB91SF6FsgWpOKPsEr1Pb6N+E6L28KI=; b=IZStnNpq1/I+ApuVpBVeH4hTznoN9Pd5vpXdIm0izXjHtNEcMzEuAYg41LcLk9C0+C 4RaxVzekvwiR+yHdBedERs25Xriog5muJpUH5K6JdeX9j4F6uIxI1HnTzLwzXVy4GixD yO68+I45kj3k99yhSaNpqzHD9Fp3KVDrGCmF0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uk0/1LocoDticB91SF6FsgWpOKPsEr1Pb6N+E6L28KI=; b=RQGiZezcB9BBZsK8+9sNskXK+FHFONpyRhd0sBks5CxcRUh3qIlLHOENUxMTMhGSqq b38E6QTfHDTSYUC6HGSuuKQgHXTeEg6ceZL3fqY2ytg7BVUU1Mln5NoHG2dtgTuS63Qs LDcsPUzYjeRSlqEMCXwaIakSWgXjw3Y04nAkdlnNtf7DtzqZI6PaVDVw98Ykb9kEdu65 IZYfA/kfwCFTdeY3OPFxWiNswVCPYm19SC2vlaumBh4y9t7ErzXE9o1dG7MKIFK8gFRC pDYRO9SOscgVZovV2ehjN5NSl0ayjn2cJF+fDR67Uos7J8S8Vs6dgZsUvVhMRBU6tnAh CdPA== X-Gm-Message-State: AOAM533dpi5sNfZrB+z4Qf3ReSs77L4GX4fC9qudXXfGTZ2BXoa6VXvx X3Lu82tx9jE7JyInccK74dvaQEB/Q/gjuA== X-Google-Smtp-Source: ABdhPJyQA+JEIQNoei+K1ch/V+Zx/sc+/gT6R6NcnBEpYJ9MDfdGQ4iDqBvG3w0PI692mwod3mWynw== X-Received: by 2002:aa7:9281:0:b029:1ec:48b2:811c with SMTP id j1-20020aa792810000b02901ec48b2811cmr11326584pfa.18.1615831360251; Mon, 15 Mar 2021 11:02:40 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id j26sm13898829pfa.35.2021.03.15.11.02.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Mar 2021 11:02:39 -0700 (PDT) From: Kees Cook To: Thomas Gleixner Cc: Kees Cook , Elena Reshetova , x86@kernel.org, Andy Lutomirski , Peter Zijlstra , Catalin Marinas , Will Deacon , Mark Rutland , Alexander Potapenko , Alexander Popov , Ard Biesheuvel , Jann Horn , Vlastimil Babka , David Hildenbrand , Mike Rapoport , Andrew Morton , Jonathan Corbet , Randy Dunlap , kernel-hardening@lists.openwall.com, linux-hardening@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 6/6] lkdtm: Add REPORT_STACK for checking stack offsets Date: Mon, 15 Mar 2021 11:02:29 -0700 Message-Id: <20210315180229.1224655-7-keescook@chromium.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210315180229.1224655-1-keescook@chromium.org> References: <20210315180229.1224655-1-keescook@chromium.org> MIME-Version: 1.0 X-Patch-Hashes: v=1; h=sha256; g=fb815901a1ccc1d9c4ca5c3e3cd3729b7f382fe2; i=b69wRsxT78r/3tM1mGa7N6ME6+rlXyFg15giRWRwPAQ=; m=aFqgiEE+nAZdug79A1F+fVTg9ZceUb0WPE8cbHqssVg=; p=ZQ32/kILkW5AD3nBZHO0VMTp4prIPkm7+DdhCHX8KdA= X-Patch-Sig: m=pgp; i=keescook@chromium.org; s=0x0x8972F4DFDC6DC026; b=iQIzBAABCgAdFiEEpcP2jyKd1g9yPm4TiXL039xtwCYFAmBPoTUACgkQiXL039xtwCaMtg//W3C jkHw9FCj0GjqFApbcxCL3jl2YwAAjXV58G4rDQYeWzbKEMXoLxCT/GGn17D3xXysAm9/dW2wqXTwB DG47+LfnvFEwMr8RG/1Z5TwURvUpKzo4uwvMAFeALrh34TQr8FiHicEsnh7Zae9vlUtfOkpjOA0vk oWbnFMRoe2jSxYPdZZWVAJAbGIfnrwvaALfb5oL+LyeL8u9+7EEUHGIVMp/HlXNDGQcTYwzqUwi/W IlCP4WjLrZDgvazHIfc7eY8ZGfvY6ZX7SGPb1u5ybkYUnEpCZrBY8RHqYDX244saxallSVPgEheQu 7q/IEyY2F5UKVUFi1R/O83XIrt6Jn35gkynezZltDe1PxA981dpO/TA3pJ2tTRW+p0F4kEIKWaCmM mF77WJRwHkMXTwKX1ML6GCELxA1bznMv4TRKyDgplEklsd4I2jZ40a2nFTBoucQzVW9f8zmYcjHT8 JEoRplIfyQytFfxYLXw6aP3v/1tYFd+I2VfHzz9c7J/e2nXELr/T2i7irF/iSqDAjeuyOLPbdiXgT sEzmOOw9gtVo6TZYCuVVUcf9tYRJPBKAe/ZMzrQRtkwaIoh2jdHhlIKiyROpgq1wfNdpT02SXsE8K FItAad3A/9tYuy56EOrjN/EilPKLKKCvHuOFdIOMXqVOAyqGJVMN2C1mhd7p+ziM= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210315_180805_700397_B8F0A500 X-CRM114-Status: GOOD ( 15.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For validating the stack offset behavior, report the offset from a given process's first seen stack address. A quick way to measure the entropy: for i in $(seq 1 1000); do echo "REPORT_STACK" >/sys/kernel/debug/provoke-crash/DIRECT done offsets=$(dmesg | grep 'Stack offset' | cut -d: -f3 | sort | uniq -c | sort -n | wc -l) echo "$(uname -m) bits of stack entropy: $(echo "obase=2; $offsets" | bc | wc -L)" Signed-off-by: Kees Cook --- drivers/misc/lkdtm/bugs.c | 17 +++++++++++++++++ drivers/misc/lkdtm/core.c | 1 + drivers/misc/lkdtm/lkdtm.h | 1 + 3 files changed, 19 insertions(+) diff --git a/drivers/misc/lkdtm/bugs.c b/drivers/misc/lkdtm/bugs.c index 110f5a8538e9..0e8254d0cf0b 100644 --- a/drivers/misc/lkdtm/bugs.c +++ b/drivers/misc/lkdtm/bugs.c @@ -134,6 +134,23 @@ noinline void lkdtm_CORRUPT_STACK_STRONG(void) __lkdtm_CORRUPT_STACK((void *)&data); } +static pid_t stack_pid; +static unsigned long stack_addr; + +void lkdtm_REPORT_STACK(void) +{ + volatile uintptr_t magic; + pid_t pid = task_pid_nr(current); + + if (pid != stack_pid) { + pr_info("Starting stack offset tracking for pid %d\n", pid); + stack_pid = pid; + stack_addr = (uintptr_t)&magic; + } + + pr_info("Stack offset: %d\n", (int)(stack_addr - (uintptr_t)&magic)); +} + void lkdtm_UNALIGNED_LOAD_STORE_WRITE(void) { static u8 data[5] __attribute__((aligned(4))) = {1, 2, 3, 4, 5}; diff --git a/drivers/misc/lkdtm/core.c b/drivers/misc/lkdtm/core.c index b2aff4d87c01..8024b6a5cc7f 100644 --- a/drivers/misc/lkdtm/core.c +++ b/drivers/misc/lkdtm/core.c @@ -110,6 +110,7 @@ static const struct crashtype crashtypes[] = { CRASHTYPE(EXHAUST_STACK), CRASHTYPE(CORRUPT_STACK), CRASHTYPE(CORRUPT_STACK_STRONG), + CRASHTYPE(REPORT_STACK), CRASHTYPE(CORRUPT_LIST_ADD), CRASHTYPE(CORRUPT_LIST_DEL), CRASHTYPE(STACK_GUARD_PAGE_LEADING), diff --git a/drivers/misc/lkdtm/lkdtm.h b/drivers/misc/lkdtm/lkdtm.h index 5ae48c64df24..99f90d3e5e9c 100644 --- a/drivers/misc/lkdtm/lkdtm.h +++ b/drivers/misc/lkdtm/lkdtm.h @@ -17,6 +17,7 @@ void lkdtm_LOOP(void); void lkdtm_EXHAUST_STACK(void); void lkdtm_CORRUPT_STACK(void); void lkdtm_CORRUPT_STACK_STRONG(void); +void lkdtm_REPORT_STACK(void); void lkdtm_UNALIGNED_LOAD_STORE_WRITE(void); void lkdtm_SOFTLOCKUP(void); void lkdtm_HARDLOCKUP(void);