From patchwork Thu Oct 11 15:14:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636845 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C06E0112B for ; Thu, 11 Oct 2018 15:20:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF0FD2B9A6 for ; Thu, 11 Oct 2018 15:20:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC66F2B9F0; Thu, 11 Oct 2018 15:20:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9546C2B9A6 for ; Thu, 11 Oct 2018 15:20:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE9C56B0008; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C5826B0010; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 672586B000A; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 1B5B66B0008 for ; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id n81-v6so8103152pfi.20 for ; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=kq1s4ljn8ySoNNOVxya1bPQFU+a5eZQhitgpvlmsRTU=; b=m/GW6B+6KU4aIBjrog31VxWc9kzF0T4D1pPou3VsFe3yosge2nII+cghsGVDMFF7k5 fZnrcIarSJEULOn6XadraKNklRFlFLutp3ug5FNdrxrho9JRNZTdREw4EHDils+hr90v BCSwGogP/ClQ6ktYUz00iwuXtm3GwA/724q39SdAWTmvFMGeQecQ2R3l6VgHAR4rqiYa HLDYCni3PNapixwd938SGwX+eLG32410cqz68OoSWy2iP5xKjvqQKqH+Alx9S4fyvSTD FR5VSTYsEup2eu0hVrQ5Sei6ZfLAlOdHXOndgENeAIyA6ASdJhUAucSsqg6566z4gn0m 30Iw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoiNQcJm+r7LgIzVeWU7ksOQrS3MhgVDmCDnsXZMgeac0ypnl/Aj MgOi0stsFogL/NMBTpPumNxEbbCHxJDNqY2+DNWHXdWliExZZPEJbJ96ueocZu4dGgAgggQHjDp vENtND+CzHRZZHUrNjtMf01Pk+aseDGb4lcmGwrJEepPBBNJ6OHllA6vXk55SYtLFDQ== X-Received: by 2002:a62:d841:: with SMTP id e62-v6mr2026139pfg.60.1539271246773; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) X-Google-Smtp-Source: ACcGV63F7MeTrIWLpl375SgIHrtENpeTEHpk0SnS3yaypHI4Kgq+/ZdmpwT/4fov7UmIRGzONeTM X-Received: by 2002:a62:d841:: with SMTP id e62-v6mr2026075pfg.60.1539271245652; Thu, 11 Oct 2018 08:20:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271245; cv=none; d=google.com; s=arc-20160816; b=rqR0jnkIpiQyEONP5UwtK7soUxvldnmlqSBv5psd3fKTkLDNfakHs7wVJ1MXUnOzsi 94YjUqZ31vVqP7rDqx7U/fM0xdtr/7NwP1iYmmLu5fN9IT88TxTdJZhBFgNi75X7jchE ceHOr1Hi0bppcrsvy5mCzcBB1lvFU4uZWKHvNuhKIG+ROQo2YY7aqDEo7z7t8FKhfmbZ HFoW/XHJNRYb9l3fytpQRqpzX3vEtDbLLFE842eXwQha3YEZUHbwI+qwkV4/lozyNsQt XGyuf2P0R3R/Ok6WLUHFSMBtgVLnDytwol8NNY2H9CSxvzokmHL13OgqjLEbQuQageek tbqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=kq1s4ljn8ySoNNOVxya1bPQFU+a5eZQhitgpvlmsRTU=; b=0LMNKB06fGE+K0k1MQsaSxC1zizivzDKwgBYbfvaWiZFBak+gz1STevwEGRzf+bRLi LPoT7r/z/oil4NT9+TjgVjCZ1LgHVQ0S22PjTwKdAG46i/2XM/WJX5L2pvvszUSplYlC q0YqwHJ3jwlGNmuzzaAEX/sodYXYXzb3eofUeFxuFg7sJSwaI8GpCSgUWyo1DVtkwuw6 u9m5cNaTTLDm9bTZRbJ0pL/aBQW5icZb1+FH0ufmMP7EsUaG3LiQMz8AcSVWAqRcRMMe jWjOyW4FqyQSZ4Q/NehlbhAGPtV+rkqBcca316nk2MW51qZpKCCM3PDF55nssqbqhLOa yAmg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019075" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 01/27] x86/cpufeatures: Add CPUIDs for Control Flow Enforcement Technology (CET) Date: Thu, 11 Oct 2018 08:14:57 -0700 Message-Id: <20181011151523.27101-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add CPUIDs for Control Flow Enforcement Technology (CET). CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking Signed-off-by: Yu-cheng Yu Reviewed-by: Borislav Petkov --- arch/x86/include/asm/cpufeatures.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 89a048c2faec..142b15da06fd 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -321,6 +321,7 @@ #define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ @@ -341,6 +342,7 @@ #define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */ #define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */ #define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */ +#define X86_FEATURE_IBT (18*32+20) /* Indirect Branch Tracking */ #define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */ #define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */ #define X86_FEATURE_FLUSH_L1D (18*32+28) /* Flush L1D cache */ From patchwork Thu Oct 11 15:14:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA700112B for ; Thu, 11 Oct 2018 15:21:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2F1D2BA26 for ; Thu, 11 Oct 2018 15:21:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A697D2BA11; Thu, 11 Oct 2018 15:21:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72DC72BA11 for ; Thu, 11 Oct 2018 15:21:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 223316B026B; Thu, 11 Oct 2018 11:20:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F2B916B0270; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FA246B000E; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 0F30E6B026E for ; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id 25-v6so4679659pfs.5 for ; Thu, 11 Oct 2018 08:20:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=V++wpL83d17DV1eWW0NnA40zJzg2Gr5sP3gX225DVI0=; b=FdXCIKbjYUlOTPc7VG5MSQQHSSTIgBN4KaMScsy0BxhD6RE1hO0bDen1jPaFXaqrjb b9Emnhb4152NMGKrB3UAqbS4tYFtRpKZcQ/Hzgg8Gz0TVhsgj9AxJ7QdBbFYK94EnLSr QdY9Jpk1hb+nKUjSMwBvAj4jDV0T9kg+qKvgA4XRa2EvibLEr6vnDaxe1CbgV5pEv1pm JxdPbXJyuPHsvMmhbxdIkQ60SwtAq4KWS9YJRuS03a9G8WbsbUGqwEMmbO4p8NIZ0UCq S0xEq3keH4AUTLwz891wFJFEnQSPhCaVsks04hyRUj6618STF55pYBmqwkxwjZlpulje mYTg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogyjs7QlVt28mOcazuhDuSAlBarxV325lQl05G3xuqpFYPtar1D K1kkMhm+RnhG92J0eBjC+LeURrzoo2UOGgRCKZCeavoDuc0zFJXDU3ZgWGyrRZcbV0evrZi3Wgq m+WZW1g65L+aSBkJcZSIBPXG/5O6EHbPNEsIAx5czs1+uMizTQdwTV8mA+Y6S8DnaPA== X-Received: by 2002:a17:902:d208:: with SMTP id t8-v6mr2049584ply.22.1539271251667; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-Smtp-Source: ACcGV61rtJn6qeFPpLpN7U+qIwp6dMf4JMzUlWOXEJwNPznHSI9q+NHdifinwATaUzDRYyr1U8jF X-Received: by 2002:a17:902:d208:: with SMTP id t8-v6mr2049245ply.22.1539271245835; Thu, 11 Oct 2018 08:20:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271245; cv=none; d=google.com; s=arc-20160816; b=zKlTcOzSRS+btH7YglAJQQYKfPkzl+SiRdvwoMcAc/236ADlxKTzNRvfFEtvGIfgEp gQRE33sL+76Fb1L7+P7N7TZ/yJgCPbKfjqTVlIgAcbqBgIB0MQpzezfJ8YH/H2sDnZw0 jxBfgLlKprr85jbSVNmfV/h7lgaWFyptfSGJVrPvcpkri/riKrNuXjMCg53jVdr0ArrX Xu9t15Aj90yEmIgpeje8OO8I6M4kKme3lzNajHujBZ5Xr0RVi4JBghUXd2dd3Htms0y/ 8Jxw6+sXc9P7nVFgIHbIOYzo5ofE+LY3KKbMiBHLsZpT1n/iScX/UtqsOgjJeZdbLCl3 NZHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=V++wpL83d17DV1eWW0NnA40zJzg2Gr5sP3gX225DVI0=; b=A+L6sV50Oi/zHSznLbDiQ4/bmCP2S61RDgco/Z3/sO3Dc1mUenfNxoSryIeVxGBfak EUk6/unf5IayFmCSn66xNjX/MD0dnlGX/NIrPnO022jxDfhNs5m3EQxfhVIkDOiYUM8F XMhNs26vM2eApVozC7jETiVjj/zxuj8nkGJXBSYTN1RBgh4UIKzgVLkHfMrR/DMcmJur 7PWnnIy1n2D+zVmbhbxBdJBiq/JmFqBDmuCTQltnn7psWd1k+4zydioJ2T6+POM2jx/P SFYJTpGRUnU4OeTUGyw9nxKaxoP2BGQAp7Qv48u28RSe3sf0p16681V3omylSJi/zUph JF1A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019079" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 02/27] x86/fpu/xstate: Change names to separate XSAVES system and user states Date: Thu, 11 Oct 2018 08:14:58 -0700 Message-Id: <20181011151523.27101-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control Flow Enforcement (CET) MSRs are XSAVES system/supervisor states. To support CET, we introduce XSAVES system states first. XSAVES is a "supervisor" instruction and, comparing to XSAVE, saves additional "supervisor" states that can be modified only from CPL 0. However, these states are per-task and not kernel's own. Rename "supervisor" states to "system" states to clearly separate them from "user" states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 4 +- arch/x86/include/asm/fpu/xstate.h | 20 +++---- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +-- arch/x86/kernel/fpu/xstate.c | 82 ++++++++++++++--------------- 6 files changed, 57 insertions(+), 61 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index a38bf5a1e37a..02c4296478c8 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -93,7 +93,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -233,7 +233,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) /* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact - * format and supervisor states in addition to modified optimization in + * format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 48581988d78c..76f83d2ac10e 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -23,15 +23,15 @@ #define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) /* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " @@ -39,7 +39,7 @@ #define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 2ea85b32421a..4bd56079048f 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -363,7 +363,7 @@ void fpu__drop(struct fpu *fpu) * Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs(void) +static inline void copy_init_user_fpstate_to_fpregs(void) { if (use_xsave()) copy_kernel_to_xregs(&init_fpstate.xsave, -1); @@ -395,7 +395,7 @@ void fpu__clear(struct fpu *fpu) preempt_disable(); fpu__initialize(fpu); user_fpu_begin(); - copy_init_fpstate_to_fpregs(); + copy_init_user_fpstate_to_fpregs(); preempt_enable(); } } diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 6abd83572b01..761c3a5a9e07 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -229,7 +229,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ u64 __init fpu__get_supported_xfeatures_mask(void) { - return XCNTXT_MASK; + return SUPPORTED_XFEATURES_MASK; } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 23f1691670b6..f77aa76ba675 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -249,11 +249,11 @@ static inline int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_ { if (use_xsave()) { if ((unsigned long)buf % 64 || fx_only) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_fxregs(buf); } else { - u64 init_bv = xfeatures_mask & ~xbv; + u64 init_bv = xfeatures_mask_user & ~xbv; if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_xregs(buf, xbv); @@ -417,7 +417,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 87a57b7642d3..e7cbaed12ef1 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -51,13 +51,14 @@ static short xsave_cpuid_features[] __initdata = { }; /* - * Mask of xstate features supported by the CPU and the kernel: + * Mask of supported 'user' xstate features derived from boot_cpu_has() and + * SUPPORTED_XFEATURES_MASK. */ -u64 xfeatures_mask __read_mostly; +u64 xfeatures_mask_user __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +83,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -113,14 +114,11 @@ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) } EXPORT_SYMBOL_GPL(cpu_has_xfeatures); -static int xfeature_is_supervisor(int xfeature_nr) +static int xfeature_is_system(int xfeature_nr) { /* - * We currently do not support supervisor states, but if - * we did, we could find out like this. - * * SDM says: If state component 'i' is a user state component, - * ECX[0] return 0; if state component i is a supervisor + * ECX[0] return 0; if state component i is a system * state component, ECX[0] returns 1. */ u32 eax, ebx, ecx, edx; @@ -131,7 +129,7 @@ static int xfeature_is_supervisor(int xfeature_nr) static int xfeature_is_user(int xfeature_nr) { - return !xfeature_is_supervisor(xfeature_nr); + return !xfeature_is_system(xfeature_nr); } /* @@ -164,7 +162,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask) == xfeatures_mask) + if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) return; /* @@ -191,7 +189,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * in a special way already: */ feature_bit = 0x2; - xfeatures = (xfeatures_mask & ~xfeatures) >> 2; + xfeatures = (xfeatures_mask_user & ~xfeatures) >> 2; /* * Update all the remaining memory layouts according to their @@ -219,20 +217,18 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) return; /* - * Make it clear that XSAVES supervisor states are not yet - * implemented should anyone expect it to work by changing - * bits in XFEATURE_MASK_* macros and XCR0. + * XCR_XFEATURE_ENABLED_MASK sets the features that are managed + * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be + * set here. */ - WARN_ONCE((xfeatures_mask & XFEATURE_MASK_SUPERVISOR), - "x86/fpu: XSAVES supervisor states are not yet implemented.\n"); - xfeatures_mask &= ~XFEATURE_MASK_SUPERVISOR; + xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -242,7 +238,7 @@ void fpu__init_cpu_xstate(void) */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask & (1UL << xfeature)); + return !!(xfeatures_mask_user & BIT_ULL(xfeature)); } /* @@ -272,7 +268,7 @@ static void __init setup_xstate_features(void) cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx); /* - * If an xfeature is supervisor state, the offset + * If an xfeature is system state, the offset * in EBX is invalid. We leave it to -1. */ if (xfeature_is_user(i)) @@ -348,7 +344,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; int i; /* @@ -421,7 +417,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = (u64)1 << 63 | xfeatures_mask; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; /* * Init all the features state with header.xfeatures being 0x0 @@ -440,8 +436,8 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports supervisor states and it uses compacted - * format. Checking a supervisor state's uncompacted offset is + * Only XSAVES supports system states and it uses compacted + * format. Checking a system state's uncompacted offset is * an error. */ if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { @@ -465,7 +461,7 @@ static int xfeature_size(int xfeature_nr) /* * 'XSAVES' implies two different things: - * 1. saving of supervisor/system state + * 1. saving of system state * 2. using the compacted format * * Use this function when dealing with the compacted format so @@ -480,8 +476,8 @@ int using_compacted_format(void) /* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ int validate_xstate_header(const struct xstate_header *hdr) { - /* No unknown or supervisor features may be set */ - if (hdr->xfeatures & (~xfeatures_mask | XFEATURE_MASK_SUPERVISOR)) + /* No unknown or system features may be set */ + if (hdr->xfeatures & ~xfeatures_mask_user) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -588,11 +584,11 @@ static void do_extra_xstate_size_checks(void) check_xstate_against_struct(i); /* - * Supervisor state components can be managed only by + * System state components can be managed only by * XSAVES, which is compacted-format only. */ if (!using_compacted_format()) - XSTATE_WARN_ON(xfeature_is_supervisor(i)); + XSTATE_WARN_ON(xfeature_is_system(i)); /* Align from the end of the previous feature */ if (xfeature_is_aligned(i)) @@ -616,7 +612,7 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask. + * Get total size of enabled xstates in XCR0/xfeatures_mask_user. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer @@ -706,7 +702,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { - xfeatures_mask = 0; + xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); } @@ -742,15 +738,15 @@ void __init fpu__init_system_xstate(void) } cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask = eax + ((u64)edx << 32); + xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue * booting without it. This is too early to BUG(). */ - pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask); + pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask_user); goto out_disable; } @@ -759,10 +755,10 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask &= ~BIT(i); + xfeatures_mask_user &= ~BIT_ULL(i); } - xfeatures_mask &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -772,9 +768,9 @@ void __init fpu__init_system_xstate(void) /* * Update info used for ptrace frames; use standard-format size and no - * supervisor xstates: + * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -782,7 +778,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask, + xfeatures_mask_user, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -801,7 +797,7 @@ void fpu__resume_cpu(void) * Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has(X86_FEATURE_XSAVE)) - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -853,7 +849,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) * have not enabled. Remember that pcntxt_mask is * what we write to the XCR0 register. */ - WARN_ONCE(!(xfeatures_mask & xstate_feature), + WARN_ONCE(!(xfeatures_mask_user & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to From patchwork Thu Oct 11 15:14:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB308679F for ; Thu, 11 Oct 2018 15:21:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 978C92B9CD for ; Thu, 11 Oct 2018 15:21:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8AF9E2B9FE; Thu, 11 Oct 2018 15:21:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 692862BA0A for ; Thu, 11 Oct 2018 15:21:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB4756B026E; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9584D6B0270; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 713646B000E; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id F13286B026D for ; Thu, 11 Oct 2018 11:20:51 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id b27-v6so8118796pfm.15 for ; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=2Ivx/y5P7nwfdFJ1sRo3cApe3vejqAWaqxVZcb0DfkY=; b=ILjt/QYPCkIsq81y5fTspSO9p1Os37mrtHL3fX5yzsg8MATHOq3D3b15qmPLXuiFuK TPDA4DqN2IWAynCVdD3h99jGe1yj5jxtwN3MbWw7m/iTcVT4sWdbPg30dgwM2CG45JRg +neg9tNYTfoGoLzu9ZvHQ7HKVOuMa0omYRUeEyAJJvQSfUy0H0j4AYh0CwFIjQRIIV+6 yl4D4guI98UOzQ/18YR7UiXzPmXCqnkruPpJc2IalOwhlaFRbs7u35ihx/wyDAfkS8vK DKTkdZtR1bpcFSAMa+A+qPy0E2kdSqUgncSyJBQGru3bGwZ78f7evXVByIR8aPQbzI3x D/3g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoiV2YZhvAqf5p+E01w6s8L2cufn0ail5M9UZFQiBsEzdJynUE80 nrxPZUlK5gPuoPuPVMos+Jb0wmo/2/8nUFPaFKBOXyqvsnOzBM/9EjWIu4neQoREG2MJA3QlrYI 6KRUG0/srQohUpHsgfcYiH0bNmP3Ni7yJGjbrV+1Ui60P+ccLxedUBTVnd5NlHsQEhA== X-Received: by 2002:a65:5bc1:: with SMTP id o1-v6mr1841010pgr.391.1539271251632; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-Smtp-Source: ACcGV60dQjYXAbTZrIRu1DbMIYemW9XCQHz5tLYivVbo5F1Rb3hdLZyBwvjHngusD7V9m/WE+UZu X-Received: by 2002:a65:5bc1:: with SMTP id o1-v6mr1840709pgr.391.1539271246027; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271245; cv=none; d=google.com; s=arc-20160816; b=SxVN4W12bbjKDlNWoMvtRpRqk2ZQsUq/TEWUJEGrYn+W7oRu4p55m+XMCyMcBq2+W3 SwrhnxOUox4m40BsQy7jIval0r61IbMt6vq8mVaA8EjFCdRIot4K93f2XBnNuEfyynjB GIIBikaZtktlq/sxKYQ9OvFKsyI2BwjabTPsBUuTPnqCyQaBITkPBr+01n+LqsFpN0uI o8bLorEzLM+dAaoVN3hZjl6VRjIZ+cZgLuu8NGCdZDW+bLBN1YRxLcR0zIAh5BugbmFu 32EgSxUgmVcLWxuap075mkKvugTxQ53QOOJzZIh8MfzeH6x5sBjssQtx+pEbxfFkWDa8 fChQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=2Ivx/y5P7nwfdFJ1sRo3cApe3vejqAWaqxVZcb0DfkY=; b=Ydu/s22tGtWrFEVjICIkwQTcHEuq5+Vqko92umil+1jJ8ji/G8Cm6Y3jdh4XvLpcQl UMyhfzNsm3c6rDGxszCYDf+jZePHwUaJmf7YZn7TL5ppenOwSqseMla0cNjdhrr6sPhv cgXy7e4OJMv97YlgLQQdYfOLLRmkp4/TGM2cRNfW2bP5GxSX6iolLWArbXtWdr4/QGxT i+vu1NBKkR7R7E0j7kdrMnzAQ9Hd7NK8SS4peQrUD2w/OQJsDuE915C3d5QVT1edOk1M +CH77hqnmy8H+USOfFCLAx+E1UAFyuaMlcapIOCJNXaz8Gee0zSRQgBFFDMXYgOQ49w2 j5Dw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019082" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 03/27] x86/fpu/xstate: Introduce XSAVES system states Date: Thu, 11 Oct 2018 08:14:59 -0700 Message-Id: <20181011151523.27101-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control Flow Enforcement (CET) MSRs are XSAVES system states. To support CET, we introduce XSAVES system states first. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 3 +- arch/x86/include/asm/fpu/xstate.h | 4 +- arch/x86/kernel/fpu/core.c | 6 +- arch/x86/kernel/fpu/init.c | 10 ---- arch/x86/kernel/fpu/xstate.c | 86 ++++++++++++++++++----------- 5 files changed, 62 insertions(+), 47 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 02c4296478c8..9a5db5a63f60 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -45,7 +45,6 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system(struct cpuinfo_x86 *c); extern void fpu__init_check_bugs(void); extern void fpu__resume_cpu(void); -extern u64 fpu__get_supported_xfeatures_mask(void); /* * Debugging facility: @@ -93,7 +92,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 76f83d2ac10e..d8e2ec99f635 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,9 +19,6 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) - /* All currently supported features */ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ @@ -40,6 +37,7 @@ #endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 4bd56079048f..5f076f593fa9 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -365,8 +365,12 @@ void fpu__drop(struct fpu *fpu) */ static inline void copy_init_user_fpstate_to_fpregs(void) { + /* + * Only XSAVES user states are copied. + * System states are preserved. + */ if (use_xsave()) - copy_kernel_to_xregs(&init_fpstate.xsave, -1); + copy_kernel_to_xregs(&init_fpstate.xsave, xfeatures_mask_user); else if (static_cpu_has(X86_FEATURE_FXSR)) copy_kernel_to_fxregs(&init_fpstate.fxsave); else diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 761c3a5a9e07..eaf9d9d479a5 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -222,16 +222,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* - * Find supported xfeatures based on cpu features and command-line input. - * This must be called after fpu__init_parse_early_param() is called and - * xfeatures_mask is enumerated. - */ -u64 __init fpu__get_supported_xfeatures_mask(void) -{ - return SUPPORTED_XFEATURES_MASK; -} - /* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch(void) { diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index e7cbaed12ef1..605ec6decf3e 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -56,9 +56,14 @@ static short xsave_cpuid_features[] __initdata = { */ u64 xfeatures_mask_user __read_mostly; +/* + * Combined XSAVES system and user states. + */ +u64 xfeatures_mask_all __read_mostly; + static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_all)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -83,7 +88,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -162,7 +167,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) + if ((xfeatures & xfeatures_mask_all) == xfeatures_mask_all) return; /* @@ -217,7 +222,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_all) return; /* * XCR_XFEATURE_ENABLED_MASK sets the features that are managed @@ -225,20 +230,19 @@ void fpu__init_cpu_xstate(void) * set here. */ - xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; - cr4_set_bits(X86_CR4_OSXSAVE); xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + /* + * MSR_IA32_XSS sets which XSAVES system states to be managed by + * XSAVES. Only XSAVES system states can be set here. + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_all & ~xfeatures_mask_user); } -/* - * Note that in the future we will likely need a pair of - * functions here: one for user xstates and the other for - * system xstates. For now, they are the same. - */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask_user & BIT_ULL(xfeature)); + return !!(xfeatures_mask_all & BIT_ULL(xfeature)); } /* @@ -344,7 +348,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_all)*8]; int i; /* @@ -417,7 +421,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_all; /* * Init all the features state with header.xfeatures being 0x0 @@ -440,7 +444,7 @@ static int xfeature_uncompacted_offset(int xfeature_nr) * format. Checking a system state's uncompacted offset is * an error. */ - if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { + if (~xfeatures_mask_user & BIT_ULL(xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -612,15 +616,12 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask_user. + * Get total size of enabled xstates in XCR0 | IA32_XSS. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer * that we use 'XSAVES' on, we could potentially overflow the * buffer because 'XSAVES' saves system states too. - * - * Note that we do not currently set any bits on IA32_XSS so - * 'XCR0 | IA32_XSS == XCR0' for now. */ static unsigned int __init get_xsaves_size(void) { @@ -702,6 +703,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { + xfeatures_mask_all = 0; xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); @@ -717,6 +719,8 @@ void __init fpu__init_system_xstate(void) static int on_boot_cpu __initdata = 1; int err; int i; + u64 cpu_user_xfeatures_mask; + u64 cpu_system_xfeatures_mask; WARN_ON_FPU(!on_boot_cpu); on_boot_cpu = 0; @@ -737,10 +741,23 @@ void __init fpu__init_system_xstate(void) return; } + /* + * Find user states supported by the processor. + * Only these bits can be set in XCR0. + */ cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask_user = eax + ((u64)edx << 32); + cpu_user_xfeatures_mask = eax + ((u64)edx << 32); + + /* + * Find system states supported by the processor. + * Only these bits can be set in IA32_XSS MSR. + */ + cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx); + cpu_system_xfeatures_mask = ecx + ((u64)edx << 32); - if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + xfeatures_mask_all = cpu_user_xfeatures_mask | cpu_system_xfeatures_mask; + + if ((xfeatures_mask_all & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -755,10 +772,11 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask_user &= ~BIT_ULL(i); + xfeatures_mask_all &= ~BIT_ULL(i); } - xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_all &= SUPPORTED_XFEATURES_MASK; + xfeatures_mask_user = xfeatures_mask_all & cpu_user_xfeatures_mask; /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -770,7 +788,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -778,7 +796,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask_user, + xfeatures_mask_all, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -798,6 +816,12 @@ void fpu__resume_cpu(void) */ if (boot_cpu_has(X86_FEATURE_XSAVE)) xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * Restore IA32_XSS + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_all & ~xfeatures_mask_user); } /* @@ -847,9 +871,9 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) /* * We should not ever be requesting features that we * have not enabled. Remember that pcntxt_mask is - * what we write to the XCR0 register. + * what we write to the XCR0 | IA32_XSS registers. */ - WARN_ONCE(!(xfeatures_mask_user & xstate_feature), + WARN_ONCE(!(xfeatures_mask_all & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -999,7 +1023,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1083,7 +1107,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1176,7 +1200,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: @@ -1232,7 +1256,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: From patchwork Thu Oct 11 15:15:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636847 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30512112B for ; Thu, 11 Oct 2018 15:20:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F3A92B9F7 for ; Thu, 11 Oct 2018 15:20:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 132582B9E4; Thu, 11 Oct 2018 15:20:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A6812B9EB for ; Thu, 11 Oct 2018 15:20:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A9DE6B0266; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F05276B0010; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D006C6B026A; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 784A86B000C for ; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id x2-v6so6204300pgr.8 for ; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mqnInkX0o99heuSorhgSZCtH9gGqlqzlH5YmGzrOXvk=; b=pAeYxYHLMDskfhnk7e8z/ZaTUzlD0SoPDfLtoE7fQ7RcF8oH7lS3KRP8HYI4Ay+97O YEpwUkyBWCiQiD3qAJuLOSqzngmLuWYPOR8O8njpcz7s4UFmxnK1qV/z9BnPCO0Ho9Rz jUq8nB+FEO1MQu4Vu/z3f74fk/r5+A/2VhHJ0mL8BfWDkaM8y3Dl0x1SPksxoX61YC+e xl+Adit6YvpM23fHvpELMcfX3nLN6TizG0C9zMeRq/OpaPUXE/W9OlegPVqf7ixp8hqN y0R0ewfe2L2AcpCecp9JKzQHaleSWVfy7k9SbZ2ZGO4sZQxVCJhnyEUEJ3J1bsD8sVDj nTiw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojIzax9Wko8Yl+ABLYQfVvAAMb/fRXDd4gOfONKiZ5XmMrOjMlE rgJ0+ZwXIb1BBvYlZQWwkJc12+M1uZjEqEPGt1oOYj/gKrj6+frFMIAQzAjKiiRI7jE5j+5ayar Gz1V+jHCCvMIRIA4RinbsRobh/S9FEQvuuN9HZvVH9XJ3Koeo9gvJIrzJNiAj5mYfJw== X-Received: by 2002:a17:902:7b94:: with SMTP id w20-v6mr2021098pll.56.1539271247165; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV635YXZLanhRyId3fxHHOj9GNZXYm1CseufpNi5nuuEsGyjVZu561MCP4a2EwCt+v8zGM0he X-Received: by 2002:a17:902:7b94:: with SMTP id w20-v6mr2021051pll.56.1539271246238; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=EUQV8EavZMNCcUIvIYFaKu2icN07DtjXOsq+4REnq7rwm2TRGDaAXksULBiyg/g3nU YmA1BfBDdMEsWS3rGkS/1LdcM06K+PXTAO01aJCS/dWYvfkouAXnsUSOjKAyKQdJkFHN 2o14GB0DRDtSwCtI0RA4PnwartWVi95Zvr+V0BW5nCLvPWKWZ4EK49yfbobSGIjSKKmL MNxWld3ZhybpKPgR34FoDde7t0XEZMIMBhItiSvLc0KQbgHYhB/PDbdCGTQRVIGuYnoX CRJ+TUovC4/EusG85l22rW8LrrWbUrrSuYTs8IvpwLsvOt+nN17ytu8D0BOTRSwcjAEF Iv9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=mqnInkX0o99heuSorhgSZCtH9gGqlqzlH5YmGzrOXvk=; b=Cha0DBMTkcu4hUIADw4SlEJZ2j5aXzShEKjYxZBkH3U6RKvrgWC5ouRuG364tF/vls yTd//aq5FIUamk/wOFYuJ2eyLm4OxWwLhomxpwen3uM8RfnTq8p2AbsfCN0c55mn1vT5 G0Z9lOlFY4lI7px+jGc61s0PP9PiKxaXRJLNhy6rSH3jA4wMLob9brwY4h2l4n43xdGZ esUqxx1eUr6ZIuIWiUwMvcTRK7hz8nVSD8jjiidWhjf4RulCQAvbgXeLMNkcXBTya5xa TdULXBHuGFHZQd1l9EYBXkvObwbRY+9R8QCor2UBMesv5h3HerN0DQUu42txoydoN6AM Zccw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019085" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 04/27] x86/fpu/xstate: Add XSAVES system states for shadow stack Date: Thu, 11 Oct 2018 08:15:00 -0700 Message-Id: <20181011151523.27101-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Intel Control-flow Enforcement Technology (CET) introduces the following MSRs into the XSAVES system states. IA32_U_CET (user-mode CET settings), IA32_PL3_SSP (user-mode shadow stack), IA32_PL0_SSP (kernel-mode shadow stack), IA32_PL1_SSP (ring-1 shadow stack), IA32_PL2_SSP (ring-2 shadow stack). Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index 202c53918ecf..e55d51d172f1 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM, XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, + XFEATURE_RESERVED, + XFEATURE_SHSTK_USER, + XFEATURE_SHSTK_KERNEL, XFEATURE_MAX, }; @@ -128,6 +131,8 @@ enum xfeature { #define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM) #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_SHSTK_USER (1 << XFEATURE_SHSTK_USER) +#define XFEATURE_MASK_SHSTK_KERNEL (1 << XFEATURE_SHSTK_KERNEL) #define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE) #define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \ @@ -229,6 +234,23 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control flow Enforcement user states + */ +struct cet_user_state { + u64 u_cet; /* user control flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +} __packed; + +/* + * State component 12 is Control flow Enforcement kernel states + */ +struct cet_kernel_state { + u64 kernel_ssp; /* kernel shadow stack */ + u64 pl1_ssp; /* ring-1 shadow stack */ + u64 pl2_ssp; /* ring-2 shadow stack */ +} __packed; + struct xstate_header { u64 xfeatures; u64 xcomp_bv; diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index d8e2ec99f635..18b60748a34d 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -28,7 +28,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_SHSTK_USER | \ + XFEATURE_MASK_SHSTK_KERNEL) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..25311ec4b731 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -130,6 +130,8 @@ #define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT) #define X86_CR4_PKE_BIT 22 /* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23 /* enable Control flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) /* * x86-64 Task Priority Register, CR8 diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 605ec6decf3e..ad36ea28bfd1 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -35,6 +35,9 @@ static const char *xfeature_names[] = "Processor Trace (unused)" , "Protection Keys User registers", "unknown xstate feature" , + "Control flow User registers" , + "Control flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = { @@ -48,6 +51,9 @@ static short xsave_cpuid_features[] __initdata = { X86_FEATURE_AVX512F, X86_FEATURE_INTEL_PT, X86_FEATURE_PKU, + 0, /* Unused */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_KERNEL */ }; /* @@ -312,6 +318,8 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_ZMM_Hi256); print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); + print_xstate_feature(XFEATURE_MASK_SHSTK_USER); + print_xstate_feature(XFEATURE_MASK_SHSTK_KERNEL); } /* @@ -558,6 +566,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_KERNEL, struct cet_kernel_state); /* * Make *SURE* to add any feature numbers in below if From patchwork Thu Oct 11 15:15:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636863 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C23A5112B for ; Thu, 11 Oct 2018 15:21:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEF2A2B9CD for ; Thu, 11 Oct 2018 15:21:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A2B022B9AA; Thu, 11 Oct 2018 15:21:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0EDF2B9FD for ; Thu, 11 Oct 2018 15:21:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE2716B026A; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8F3816B0271; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 562456B026F; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id CA0726B026A for ; Thu, 11 Oct 2018 11:20:51 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id z12-v6so8089441pfl.17 for ; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=B6jB26Wq+R48Psj4oCvVnmS8X5ABA+QSLb1HTxWzBzY=; b=HruymcRHR00I3cAp24UJ0O8435CnKURlZvk3e55fM3hRCQV5vSmIXtBawi7/tBH+C0 AAL/KAW+hCR/eoSTtJ4Xj3OvBMRluSK/7DyXFkHa+Ym6RpP3sVwKRQloRjk4k9mH+nu5 GA8MkihtEdplSP2+QjNcbLisX0iEw8sftypW0KClVQihSNmUJKRXFJEjhHTKIma5GCTO SL7RrouGvrxSSQlfHt0PSwulLMI5lMeyiW6NJRr+TqeyvYfjfZ29F1c3Q7oXBOkUgZLY slFMyIUVl8KG/k4+CCR8lDWNVoaB0WW7Ml/udPMlnHpE1yyfPXNtvpb8x6esh+hPf4Rw CH3w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohF65vCX4YM/sVN3LlSzbt+8IW1s01tmHRFVgD2olTd/IvP0cre FayNeNEinNJIuYWbsgwyjPdUXTg28jYDsxIBIdNUR2+DjoEv31dqZOlHEMBWMVv62UN2VgyRL8Y xpymEKlmkWnQFTttp3GcaaS8kHSNZArFfueYdf7nCgGJtMyab1IQImV+IXHaLTi6JOQ== X-Received: by 2002:a63:af5b:: with SMTP id s27-v6mr1813777pgo.448.1539271251453; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-Smtp-Source: ACcGV62ZT1dGEaNOssRd20E/jl+8ri60oxEFzqIZs1NXa2bTmePQ2KYwax0BAkBPeiIemurXAOSb X-Received: by 2002:a63:af5b:: with SMTP id s27-v6mr1813515pgo.448.1539271246364; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=cKnuhRbn47A1vCGA18iscFxqUin3Cnrtte7k0IxuhvMlle8R1mnmoOUmDdXljPrxYy tW0gBf7/UWGuM9a7+/ZYPDlRQ3ELQ6eVKQRmzAstVPw8H/JufEHD4Um8LjglKRBarZaa s3mjjXSlC9kE/Tf+EeVc6I4gETY8K1cD5V3sgWoh33JYQtmEtmOUxJA3OoigWdE1gh4H T5boYVINLn7eZzXmp2JV4MD2KvG7Z7NmFNfy3dqxXwralojY/m+JlFLNHOgTYMBOUbQd cqduoEkHomSfbWKMdC/PtCClP0wpMjxNBduQ+XSBn5WZHyd6HJsTwDijAyp8LYGfKIkn ajNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=B6jB26Wq+R48Psj4oCvVnmS8X5ABA+QSLb1HTxWzBzY=; b=rAOWw+zGIBeq0HcTjHbCrc0DjHSdirIQ46EfXoOJobRqaJvVz31Pk4O4nI/s7XIzea i/gii64PrKSlXWWDmDW2yTsj0CrMeaOCOz0BLUKi5RNO7FaUAD/lZCELpFDLRBHYlFsB s2SskNrU27woNe1k4bDRBJL9pYcVGxKoRW+K2cnyosFt0SvTfd31v60xF/9/mCT97VZT m9J7SxiCsH1GBeuIFmdhsCz/k2V4ZoZzivcvU+Fg+z8ed3b+9VoearrNs3Fd8TFw36I6 XwWJRElPK/3QsD4OJrnKFI0WIV2juEt0YiRkfr9whNtTdiEEr3uu8fCWFsnXluqcyiXM OMRQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id d35-v6si29570683pla.116.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019091" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 05/27] Documentation/x86: Add CET description Date: Thu, 11 Oct 2018 08:15:01 -0700 Message-Id: <20181011151523.27101-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Explain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters. Signed-off-by: Yu-cheng Yu --- .../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 11 + Documentation/x86/intel_cet.rst | 266 ++++++++++++++++++ 4 files changed, 284 insertions(+) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 92eb1f42240d..3854423f7c86 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. diff --git a/Documentation/index.rst b/Documentation/index.rst index 5db7e87c7cb1..1cdc139adb40 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -104,6 +104,7 @@ implementation. :maxdepth: 2 sh/index + x86/index Filesystem Documentation ------------------------ diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst new file mode 100644 index 000000000000..9c34d8cbc8f0 --- /dev/null +++ b/Documentation/x86/index.rst @@ -0,0 +1,11 @@ +======================= +X86 Documentation +======================= + +Control Flow Enforcement +======================== + +.. toctree:: + :maxdepth: 1 + + intel_cet diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst new file mode 100644 index 000000000000..946f4802a51f --- /dev/null +++ b/Documentation/x86/intel_cet.rst @@ -0,0 +1,266 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protection against +return/jump-oriented programming (ROP) attacks. It can be implemented +to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented on the 64-bit kernel. +However, 32-bit applications are supported under the compatibility +mode. + +CET includes shadow stack (SHSTK) and indirect branch tracking (IBT). +The SHSTK is a secondary stack allocated from memory. The processor +automatically pushes/pops a secure copy to the SHSTK every return +address and, by comparing the secure copy to the program stack copy, +verifies function returns are as intended. The IBT verifies all +indirect CALL/JMP targets are intended and marked by the compiler with +'ENDBR' op codes. + +There are two kernel configuration options: + + INTEL_X86_SHADOW_STACK_USER, and + INTEL_X86_BRANCH_TRACKING_USER. + +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later +are required. To build a CET-enabled application, GLIBC v2.28 or +later is also required. + +There are two command-line options for disabling CET features: + + no_cet_shstk - disables SHSTK, and + no_cet_ibt - disables IBT. + +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT. + +[2] CET assembly instructions +============================= + +RDSSP %r + Read the SHSTK pointer into %r. + +INCSSP %r + Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated + in the operand register. The GLIBC longjmp uses INCSSP to unwind + the SHSTK until that matches the program stack. When it is + necessary to unwind beyond 255 steps, longjmp divides and repeats + the process. + +RSTORSSP (%r) + Switch to the SHSTK indicated in the 'restore token' pointed by + the operand register and replace the 'restore token' with a new + token to be saved (with SAVEPREVSSP) for the outgoing SHSTK. + +:: + + Before RSTORSSP + + Incoming SHSTK Current/Outgoing SHSTK + + |----------------------| |----------------------| + addr=x | | ssp-> | | + |----------------------| |----------------------| + (%r)-> | rstor_token=(x|Lg) | addr=y-8 | | + |----------------------| |----------------------| + + After RSTORSSP + + |----------------------| |----------------------| + | | | | + |----------------------| |----------------------| + ssp-> | rstor_token=(y|Bz|Lg)| addr=y-8 | | + |----------------------| |----------------------| + + note: + 1. Only valid addresses and restore tokens can be on the + user-mode SHSTK. + 2. A token is always of type u64 and must align to u64. + 3. The incoming SHSTK pointer in a rstor_token must point to + immediately above the token. + 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK. + 5. 'Bz' is bit[1] of a rstor_token indicating the token is to + be used only for the next SAVEPREVSSP and invalid for the + RSTORSSP. + +SAVEPREVSSP + Store the SHSTK 'restore token' pointed by + (current_SHSTK_pointer + 8). + +:: + + After SAVEPREVSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) | + |----------------------| |----------------------| + +WRUSS %r0, (%r1) + Write the value in %r0 to the SHSTK address pointed by (%r1). + This is a kernel-mode only instruction. + +ENDBR + The compiler inserts an ENDBR at all valid branch targets. Any + CALL/JMP to a target without an ENDBR triggers a control + protection fault. + +[3] Application Enabling +======================== + +An application's CET capability is marked in its ELF header and can +be verified from the following command output, in the +NT_GNU_PROPERTY_TYPE_0 field: + + readelf -n + +If an application supports CET and is statically linked, it will run +with CET protection. If the application needs any shared libraries, +the loader checks all dependencies and enables CET only when all +requirements are met. + +[4] Legacy Libraries +==================== + +GLIBC provides a few tunables for backward compatibility. + +GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT + Turn off SHSTK/IBT for the current shell. + +GLIBC_TUNABLES=glibc.tune.x86_shstk= + This controls how dlopen() handles SHSTK legacy libraries: + on: continue with SHSTK enabled; + permissive: continue with SHSTK off. + +[5] CET system calls +==================== + +The following arch_prctl() system calls are added for CET: + +arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) + Return CET feature status. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = SHSTK/IBT status + *(addr + 1) = SHSTK base address + *(addr + 2) = SHSTK size + +arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) + Disable SHSTK and/or IBT specified in 'features'. Return -EPERM + if CET is locked. + +arch_prctl(ARCH_X86_CET_LOCK) + Lock in CET feature. + +arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) + Allocate a new SHSTK and put a restore token at top. + + The parameter 'addr' is a pointer to a user buffer and indicates + the desired SHSTK size to allocate. On returning to the caller, + the kernel fills *addr with the base address of the new SHSTK. + +arch_prctl(ARCH_X86_CET_GET_LEGACY_BITMAP, unsigned long *addr) + Allocate an IBT legacy code bitmap if the current task does not + have one. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = IBT bitmap base address + *(addr + 1) = IBT bitmap size + +Note: + There is no CET enabling arch_prctl function. By design, CET is + enabled automatically if the binary and the system can support it. + + The parameters passed are always unsigned 64-bit. When an ia32 + application passing pointers, it should only use the lower 32 bits. + +[6] The implementation of the SHSTK +=================================== + +SHSTK size +---------- + +A task's SHSTK is allocated from memory to a fixed size of +RLIMIT_STACK. + +Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, we can use a large SHSTK to +cover the condition that both the program stack and the sigaltstack +run out. + +The kernel creates a restore token at the SHSTK restoring address and +verifies that token when restoring from the signal handler. + +Fork +---- + +The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be +read-only and dirty. When a SHSTK PTE is not present, RO, and dirty, +a SHSTK access triggers a page fault with an additional SHSTK bit set +in the page fault error code. + +When a task forks a child, its SHSTK PTEs are copied and both the +parent's and the child's SHSTK PTEs are cleared of the dirty bit. +Upon the next SHSTK access, the resulting SHSTK page fault is handled +by page copy/re-use. + +When a pthread child is created, the kernel allocates a new SHSTK for +the new thread. + +Setjmp/Longjmp +-------------- + +Longjmp unwinds SHSTK until it matches the program stack. + +Ucontext +-------- + +In GLIBC, getcontext/setcontext is implemented in similar way as +setjmp/longjmp. + +When makecontext creates a new ucontext, a new SHSTK is allocated for +that context with ARCH_X86_CET_ALLOC_SHSTK the syscall. The kernel +creates a restore token at the top of the new SHSTK and the user-mode +code switches to the new SHSTK with the RSTORSSP instruction. + +[7] The management of read-only & dirty PTEs for SHSTK +====================================================== + +A RO and dirty PTE exists in the following cases: + +(a) A page is modified and then shared with a fork()'ed child; +(b) A R/O page that has been COW'ed; +(c) A SHSTK page. + +The processor only checks the dirty bit for (c). To prevent the use +of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as +DIRTY_SW for (a) and (b) above. This results to the following PTE +settings: + +Modified PTE: (R/W + DIRTY_HW) +Modified and shared PTE: (R/O + DIRTY_SW) +R/O PTE, COW'ed: (R/O + DIRTY_SW) +SHSTK PTE: (R/O + DIRTY_HW) +SHSTK PTE, COW'ed: (R/O + DIRTY_HW) +SHSTK PTE, shared: (R/O + DIRTY_SW) + +Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs. + +[8] The implementation of IBT +============================= + +The kernel provides IBT support in mmap() of the legacy code bit map. +However, the management of the bitmap is done in the GLIBC or the +application. From patchwork Thu Oct 11 15:15:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8723112B for ; Thu, 11 Oct 2018 15:21:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7A412BA11 for ; Thu, 11 Oct 2018 15:21:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B1342BA15; Thu, 11 Oct 2018 15:21:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3A5F2B9D8 for ; Thu, 11 Oct 2018 15:21:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04E246B000E; Thu, 11 Oct 2018 11:20:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C49EC6B026C; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F99D6B026B; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id E9E496B026C for ; Thu, 11 Oct 2018 11:20:51 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id r67-v6so8121171pfd.21 for ; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=M9x8AOZuDGMyp8ZshajdI9+XzUxPDGxQCLteaWp8pUc=; b=KUZr/KzR+4wAUKLOEsgAO0k3bmT9Mk4f4Q31g8S7+SSfowLp+021xMLks1gCWZw8Nw kOwLTvlJAQZmhr2RnGucKW7W1OmT6bBI+jlqeE/M5dcQlZWSGG6IPZO3IJoUKWqjt2cW YlWmKhbdn19kkmuz+byfhDphFKNphVXcuSQqdvCr+iru9i6i32n/eQTsQIS/rKnd3kML 0f0MOjPHaE4w5rXelw9SSkuJ1TONaZvtSbpO3Vgj8U3orDQD224MBLtYKJ8wdT5aLuSj aWGNJhKEVU4WNJUSTADcLFS6NcaEC2YKtY8InFbHNlH6sPUt33a3hH1yJykmO0ws+hYL XwmA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogFAdRvz2yxwEpZ81ydG3xklzTV35RaqwxqHVVW5CmLzNRW2F0q 33dRJCs3CqFZ4F84JxVDeHXCQGymH8jaXzFAFMhijHhm5iEgyaXyIOq/h5nqcDSZKOdJrSgjRE2 q/sPTJVr6h5gkqvu5ji96aNhA14s0b92Tj/bT39fTWzeFCy7OKDdxftGEzpvM2Nid1A== X-Received: by 2002:a17:902:1744:: with SMTP id i62-v6mr1985010pli.216.1539271251617; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-Smtp-Source: ACcGV63u8BmBHN3UXdVqPUzQpx0W/K3RJLdKxk4Ftl9vTZb5a+vZqYVDu4HaCyD6qL/2R0UTWtBr X-Received: by 2002:a17:902:1744:: with SMTP id i62-v6mr1984743pli.216.1539271246658; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=jdtNxk5CP+gkKPPw4mstbMBCcc/C0Pa45fwNXt50hBe8zxXt+BQKG3myezFydN5aOe hxBI5JVh+F87aQYywsp0recZmhRYC3e/ZcOSHdVWGp89lxgiGhz/iVUl7pbGGsOAMlqE qHpDTgoBKhe+ZLPUmy9C7+iyydb+S0GA22VBcyRVOWn2A+61c3st7Mrk4RzGbC9Ufvlw yknutfcBwJbac2GKloa5XdWIJUjVcGe5vljAEgf0Gn+cjAZ5ESdFnQhSe4AatQne3YcP OHarkXFr6I90J3C7WOzTKNtIwZByv+k6RweAsAkTHaYvaPr+aiAWeopCcpSi3oRgRN1b RYHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=M9x8AOZuDGMyp8ZshajdI9+XzUxPDGxQCLteaWp8pUc=; b=nXwzJHNAzhCbrQlMPawe/5InlqmTbAmIgciyLUHsx0EkGrOf7iIgY0qQgFdYm+cIKZ ZXuMS5VOHOVgCS1jSa6FPgv0rEhoJMb5SFGY6fFqysCkL8R4F8qyBn6kkf1XBkWTC8tu 0zuokebb2bk712lqgDzlTcUrGa05nvrbzfOBJ6oxyU4xSQ9DWoVqjMx4c0rC2Lc04q/G u0kSy9Nuaib4txeP4VT3EeNYb/hEast3FdJLpwCB5emLFbPbGipRf1L9ddgpZquT/Azn j/OA5T0fodQLWZz+cwJ6taXjghIdI8vJlPLmp341jKp/zyLW1PjOs+Oo3s72Su8OZhD8 EduA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id d35-v6si29570683pla.116.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019095" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 06/27] x86/cet: Control protection exception handler Date: Thu, 11 Oct 2018 08:15:02 -0700 Message-Id: <20181011151523.27101-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction. The control protection exception handler works in a similar way as the general protection fault handler. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 ++ arch/x86/kernel/signal_compat.c | 2 +- arch/x86/kernel/traps.c | 64 ++++++++++++++++++++++++++++++ include/uapi/asm-generic/siginfo.h | 3 +- 6 files changed, 75 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 957dfb693ecc..5f4914e988df 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1000,7 +1000,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 3de69330e6c5..5196050ff3d5 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void); @@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long); #endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *, long); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long); @@ -142,6 +144,7 @@ enum { X86_TRAP_AC, /* 17, Alignment Check */ X86_TRAP_MC, /* 18, Machine Check */ X86_TRAP_XF, /* 19, SIMD Floating-Point Exception */ + X86_TRAP_CP = 21, /* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ }; diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 01adea278a71..66ebc8cb16e2 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = { #elif defined(CONFIG_X86_32) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32), #endif + +#ifdef CONFIG_X86_64 + INTG(X86_TRAP_CP, control_protection), +#endif }; /* diff --git a/arch/x86/kernel/signal_compat.c b/arch/x86/kernel/signal_compat.c index 9ccbf0576cd0..c572a3de1037 100644 --- a/arch/x86/kernel/signal_compat.c +++ b/arch/x86/kernel/signal_compat.c @@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void) */ BUILD_BUG_ON(NSIGILL != 11); BUILD_BUG_ON(NSIGFPE != 15); - BUILD_BUG_ON(NSIGSEGV != 7); + BUILD_BUG_ON(NSIGSEGV != 8); BUILD_BUG_ON(NSIGBUS != 5); BUILD_BUG_ON(NSIGTRAP != 5); BUILD_BUG_ON(NSIGCHLD != 6); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index e6db475164ed..4188775681cf 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -578,6 +578,70 @@ do_general_protection(struct pt_regs *regs, long error_code) } NOKPROBE_SYMBOL(do_general_protection); +static const char *control_protection_err[] = +{ + "unknown", + "near-ret", + "far-ret/iret", + "endbranch", + "rstorssp", + "setssbsy", +}; + +/* + * When a control protection exception occurs, send a signal + * to the responsible application. Currently, control + * protection is only enabled for the user mode. This + * exception should not come from the kernel mode. + */ +dotraplinkage void +do_control_protection(struct pt_regs *regs, long error_code) +{ + struct task_struct *tsk; + siginfo_t info; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); + if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code, X86_TRAP_CP, SIGSEGV) == NOTIFY_STOP) + return; + cond_local_irq_enable(regs); + + if (!user_mode(regs)) + die("kernel control protection fault", regs, error_code); + + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control " + "protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + printk_ratelimit()) { + unsigned int max_err; + + max_err = ARRAY_SIZE(control_protection_err) - 1; + if ((error_code < 0) || (error_code > max_err)) + error_code = 0; + pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); + } + + clear_siginfo(&info); + info.si_signo = SIGSEGV; + info.si_errno = error_code; + info.si_code = SEGV_CPERR; + info.si_addr = (void __user*)uprobe_get_trap_addr(regs); + force_sig_info(SIGSEGV, &info, tsk); +} +NOKPROBE_SYMBOL(do_control_protection); + dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) { #ifdef CONFIG_DYNAMIC_FTRACE diff --git a/include/uapi/asm-generic/siginfo.h b/include/uapi/asm-generic/siginfo.h index 80e2a7227205..264d8e03270f 100644 --- a/include/uapi/asm-generic/siginfo.h +++ b/include/uapi/asm-generic/siginfo.h @@ -228,7 +228,8 @@ typedef struct siginfo { #define SEGV_ACCADI 5 /* ADI not enabled for mapped object */ #define SEGV_ADIDERR 6 /* Disrupting MCD error */ #define SEGV_ADIPERR 7 /* Precise MCD exception */ -#define NSIGSEGV 7 +#define SEGV_CPERR 8 +#define NSIGSEGV 8 /* * SIGBUS si_codes From patchwork Thu Oct 11 15:15:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636849 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D3C6679F for ; Thu, 11 Oct 2018 15:20:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D4612B9E4 for ; Thu, 11 Oct 2018 15:20:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8ACB92B9F7; Thu, 11 Oct 2018 15:20:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 569572B9FD for ; Thu, 11 Oct 2018 15:20:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F5246B0269; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD916B0010; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A77B6B000C; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 7E4E46B000D for ; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id f4-v6so8141245pff.2 for ; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=8LG+JGAGZT/96PIDACi1jNzfYo3k6CvhPo8ogtFgGUM=; b=qvsSv2KNP1wAvPYSonGamMbxMNhVKq0uuYeC96qEGNrdp9IG/2UXo6lo8kkbAbYqqv 112P/baQxgFRDaYsw04SNVJ6Bqk0N61vUeVaSjOW83KU/O3CFZ2MV0V6Ol1OPIGh/8ju JX30BslYCxViAQ16/s7GTffQoPDhEpkTqeS/lQyjW6pCxtigaRyJueYo4WUD2xjtLYJY shuEhsh0kOj6DCxEQc+vQrPZOIsed3FP+x8wAJl5ZCUbskpcuBz9yuIdTFQHs+BBOD9G 1neovhR/VdUq35j3Zm+a89PZJCYl2OKwf/RLDoC2UXs81ow6e1sDCMXSmOYU8JXLzYNL wb3w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogI4I0wx3YejMzqhMiUGLOTN38e2ikBmK8KoiUfVAtXXzqVgr/7 Fhz080jgWBzHAxO/XBG9C6POqBP08Sj3WfqtZhNbfXSmjzACzyaq2dRl3fezdiqjFkFwqk5e1q7 Pm7VkQD9+GxhxaoD5xzOnBPIL32RYf+AGD3vJE+AqlzSF0WokbcFiTOs0v+OXj7VP7A== X-Received: by 2002:a17:902:22cc:: with SMTP id o12-v6mr2028458plg.108.1539271247208; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV63P63VzUMEk8C9rRl5yCg4Xj1oEyoDQkp9D99NKhOIUYGPl3dltqrQFgh5Ygz4qfojRlcRo X-Received: by 2002:a17:902:22cc:: with SMTP id o12-v6mr2028412plg.108.1539271246471; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=zboSxl6UgrhJ+XEnw9eNii+fIX9f5rgpfkBUJ3N4rBERM+sDY51mofuICGdQZFkXYJ sQbxoq36AJQ2yrdVwr2p60/KHeHD7dnU0xrEpluLXOmg34NxHmVH2MvIgqXI2mrCNy9E QpKd8lih8leznO+uexwZYSzxyUf1qqcQEG0C0or+LYbClLp8Aj6xdywL9weiVC3KnzD9 lL4hlXaXp/Ec27hcaY20Vrrf+7YxsStZI0FBzyXvGIoA4OqEGV/1f1rBDLRuBPv6eVYX I//GUr0WvsL7vb3gJ9ixEx1Vs7+EvLisq2/2GUtaWlP5qyMHMmEOKfVRj2gbkXVd+Ksg xtAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=8LG+JGAGZT/96PIDACi1jNzfYo3k6CvhPo8ogtFgGUM=; b=ZW/8Y9miU0Hc4FLd6F0cVcj0em7Y0Wo37fjReOCBo7vaGMiPl0ZGA2XlRU4Mj7Nb51 otlsmyOGZrEGqnrRUthaV/xYykYnHUvf0rAg4l7GUM2KxxrfoPhRcUxty/z9611JhTDY CcPdZSk/6pU3AS0Yka9sDDKZqH2KG06648gMrs91rI4bsCdjTXhXMF8Ejf9NvSwXYk/3 UrXnAUxMiHsLQA0Nmi63+WkYqDOY2KxW85H66NAKzUWN4utuWEAUtwi5OQg8eSLrGiwp e4zs4aaDz/u73swgvoQA1in5zW2xk8vfi5shNM67a7pWa+Bvy2Mx3+ktfsipG7boDtEO fNjg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019098" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 07/27] mm/mmap: Create a guard area between VMAs Date: Thu, 11 Oct 2018 08:15:03 -0700 Message-Id: <20181011151523.27101-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Create a guard area between VMAs to detect memory corruption. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 30 ++++++++++++++++++++---------- mm/Kconfig | 7 +++++++ 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0416a7204be3..53cfc104c0fb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2417,24 +2417,34 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { unsigned long vm_start = vma->vm_start; + unsigned long gap = 0; + + if (vma->vm_flags & VM_GROWSDOWN) + gap = stack_guard_gap; + else if (IS_ENABLED(CONFIG_VM_AREA_GUARD)) + gap = PAGE_SIZE; + + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } return vm_start; } static inline unsigned long vm_end_gap(struct vm_area_struct *vma) { unsigned long vm_end = vma->vm_end; + unsigned long gap = 0; + + if (vma->vm_flags & VM_GROWSUP) + gap = stack_guard_gap; + else if (IS_ENABLED(CONFIG_VM_AREA_GUARD)) + gap = PAGE_SIZE; + + vm_end += gap; + if (vm_end < vma->vm_end) + vm_end = -PAGE_SIZE; - if (vma->vm_flags & VM_GROWSUP) { - vm_end += stack_guard_gap; - if (vm_end < vma->vm_end) - vm_end = -PAGE_SIZE; - } return vm_end; } diff --git a/mm/Kconfig b/mm/Kconfig index de64ea658716..0cdcad65640d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -764,4 +764,11 @@ config GUP_BENCHMARK config ARCH_HAS_PTE_SPECIAL bool +config VM_AREA_GUARD + bool "VM area guard" + default n + help + Create a guard area between VM areas so that access beyond + limit can be detected. + endmenu From patchwork Thu Oct 11 15:15:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 47B1817E1 for ; Thu, 11 Oct 2018 15:21:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3695F2B9BF for ; Thu, 11 Oct 2018 15:21:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A7962B9FE; Thu, 11 Oct 2018 15:21:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB35D2BA08 for ; Thu, 11 Oct 2018 15:21:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FEF16B000D; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4EED26B000E; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22BF16B000A; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id A83916B000E for ; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id f5-v6so6468235plf.11 for ; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=McdE7yR3ZVq5AQz5HphRhWAyx0AF717bYF00gYIP8+Y=; b=S8/wqDnv2OQO4aosRM0P0Dw8jEcANJ6h2GyxDAQbN+Vp6GojBpBR7a9dfUYfLxjnch FquMD8I4dQRznj6IVoqUUy1ICUT2sQjiLYAi7BnrG2Cz5yNVaSqu1RbZzSiwalO21PBs zoUUWuSfFR/9bKCC6r5Z/AFBZf89ieqmICH5bmEciGezgy8RNZMneeG/wvdvjUN0/i19 HMTjU6R/3Poc9WSzNZwqP0vaVb5S8AzfgFEqsbpmXFwlBFZyuQbouPrmmMwjKRGzbMXb Q04CUwNuNvIToVDPTZSWB1b7HF6yjj9+b7IKnnAItdnMBWYWnyLXr6cEvAc3UZRzs/Vm 0cRQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogmowKqf1NTebsV+g0RcUrP6UkYOKUAV6Dl+9ZAkZ2JcLOhMM6/ HEiq2023YyYZr+JZUAvrUaxAuYas/DDtFlmwlUUX5lgOmT3G6GkKiGyXijuRbUXIsw9KAsCwLv0 9RaXprowgmjzRSzcTBvyMdMjnao48ibrFB+LnqWfe9Q9jUkqu5dlaKsSVDk68uz+arQ== X-Received: by 2002:a17:902:5a4d:: with SMTP id f13-v6mr2019861plm.114.1539271247351; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV62RxXIB6XrEgpLZOu+a9g/RmepsJyic05+lbSb36SX+hXYG5f0f6KWYhNfaNC9fqikmlUWc X-Received: by 2002:a17:902:5a4d:: with SMTP id f13-v6mr2019830plm.114.1539271246653; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=K5Z2KKRdhBi/zoPQ+JsAnHUSvHxlNgqwAPQoxk/oJjRj0w2NNEsX4ZRd4jJkzltGOM 45tNBmo+GuX99EmcOUCzXB0KpO4ssL3UOwYzvwNoQLWFtRcXq5I+lyg/u7hEHtLZx/XX XcYRa+xQGjUsrSxwPYQKDFUv5cEqZ7kV+LRCawf0Ya5zfFZY11sp8nFUzXcg5km4IuOX 8spq04BBJwF6RfG9N0Bpfm4PsGgxDVxlv76fu1CfPdbF1FCM03s4AqGXwBdVhMFG88sw +tg7nHxkYJDtMIjesfVDRig3H0hll2hVg15CG0ExzAM0nuYsv9Ye24lxMB/bQIy6R7Ms 79jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=McdE7yR3ZVq5AQz5HphRhWAyx0AF717bYF00gYIP8+Y=; b=WBdihsHmHayv+39ESHJY9jl/D7myJNslSrBIP5dpHkBQtc6HAIWTt+nQGfUfN6tdnb Z9hPLsfNDjKz/RumG48sKFZBn/N/PERPKd+aKd5g4sG44Q9ZpnAs+AhSrP+6J6gRo4JO e+r/c+KcVRWHJpUGqg5SY1EInF0lON6ee0nYpGZn89uCyTXXlqbvi64Rde02VYEzYWIM dE0F/F0waV4sAXwsRkIQsE132BL9aOD3q0LynMKzbCdrSG4qFoIWmGdXdsItrhWl5GFz BvKDs88Ia4yjmGYmEllW1K5Rh2WsQvd+3fyAR92JGyZPZMws9r2FMcRGvHL1LS2W2F3W ktTw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019102" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:44 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 08/27] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Date: Thu, 11 Oct 2018 08:15:04 -0700 Message-Id: <20181011151523.27101-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER. An application has shadow stack protection when all the following are true: (1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled, (2) The running processor supports the shadow stack, (3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack. If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection. The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 26 ++++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 33 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..5f73335b7a3a 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1913,6 +1913,32 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_INTEL_CET + select ARCH_HAS_SHSTK + select VM_AREA_GUARD + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 8f6e7eb8ae9f..b28842b80295 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -152,6 +152,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Thu Oct 11 15:15:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1348C17E1 for ; Thu, 11 Oct 2018 15:21:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 023002B9BF for ; Thu, 11 Oct 2018 15:21:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E6C8C2B9F1; Thu, 11 Oct 2018 15:21:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74F282B9F1 for ; Thu, 11 Oct 2018 15:21:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 721076B026D; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 65FDA6B026A; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E3966B026B; Thu, 11 Oct 2018 11:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id C82A26B000E for ; Thu, 11 Oct 2018 11:20:51 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id m4-v6so6230336pgv.15 for ; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=D27RwH8NeKlKuxHKwLxQrrM0B4XZoNvu8mXVfNenXUU=; b=HzD/+1/PiB2iob9szw2ZypN5pcxaTehF9ZypL89RQ9zSHUP0YhPtebMPnUe3HvNBsD nL+0+NwzfWPzLdxmIEFnhpjt7VdmhPyHqLvbCutSOKth4vG+JaX0k6eq4NzZXScZE/sh 7QdXUK/2xztUwi52aM1Igk+za9bDe70ujcOaZxF9QRbOQJK0g4QNPU0GI3l2bPUyfkb7 0Ckpq/DF0RJ9CY5/UFcKdMyzoAzR6bgSENXR0n202XvDxwkeZPqzj0xESbJnMYfHKL1T OkuUPMZCIWyMtuaJJT2s1lEX/GK+3ALq1xsNucZAtybqe1C6Rd/lJXELOm1QKwQxlRLB PFbA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohjayrNQd0K9b8AjhsjTfZScvM1NDxdiLK1RyrdctnWRDUCz6uu EgkIAqtxx+rrOB2QuZoPbdo9X6gkGd0eSEAstT5xSPXLY26ycJ/bGUTZiDm7jpRIqYRepzmpGxj ODVyDb7PzilESD/oFnAb2WCAlkC+eqjnvpC/vLgSF6rSBhXOuH+6ep3xodSEO9LhhEw== X-Received: by 2002:a17:902:d01:: with SMTP id 1-v6mr2055167plu.88.1539271251507; Thu, 11 Oct 2018 08:20:51 -0700 (PDT) X-Google-Smtp-Source: ACcGV62S1Ln4or9vNkLNkXXkpJfGoHQvI4pcMXkYheeASz8NZh5LFNWwkRaUDPuvnfngLBZIr026 X-Received: by 2002:a17:902:d01:: with SMTP id 1-v6mr2054920plu.88.1539271246877; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=DNMubwIfIoN84SkwOqEKWIfKCyPyRnX69mZ1pgciNJQcJ/+NSbQ+WqYbMNJlntGWgp 7aQSuYT63x7h5gJqJudgrBiH9uCQHVb8lc5A+8n917Bp5yqcxNuiJP7fPz3r2xS56O+y dbPS76Av2Q1dqRbHq36IfHDdfxUUCO3F+ktfsK3Zj0lk9tLYJhK4nE0+rFHwSxMARD6K 0YYhixaAOrFQkbJ+17PAjmVP2hLWO25UiJcJQXIr0NaSmvBT5ff4+KOLmTN2nDnQ6HBN SlTTzr6JYvG9koY663CnWHs0lbuDYAhhYUARLZCqX+Dq37haBy/mwnRP1wemKaFAvJBE uc7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=D27RwH8NeKlKuxHKwLxQrrM0B4XZoNvu8mXVfNenXUU=; b=kxGnRlWB/haVDgI3ArryCrLNc3w5kfTPeS5MyE5sUrsVakCkOT0i17Rac5aPLgmPLH ukFCSSLroGnaUU9rk8hOh25sZOFdNj2bHCRRPQZ/Cc3s7mTo1WqEQJGE3yK2FzaK7aLc OHgZBTnBjlB19IvtSe6s2nX1df3zQwfNSvJTLjf22uwraoCNqiOo3VmaTiYTFhpliRv7 lwA+2/ifgJfI4PSxTR2/kekW+siJMc6kocVUKwhyeIA+uTjl45fybtPqcAJwqpbIviqN 9nooIHr++vIfa0laEQDmI+obONh7nRkoxcVwGp/br2IU2vLrcbKh27ZShIZ6+YlhclsY yh9Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id m7-v6si30485701pfi.286.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019105" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:44 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 09/27] mm: Introduce VM_SHSTK for shadow stack memory Date: Thu, 11 Oct 2018 08:15:05 -0700 Message-Id: <20181011151523.27101-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP VM_SHSTK indicates a shadow stack memory area. The shadow stack is implemented only for the 64-bit kernel. Signed-off-by: Yu-cheng Yu --- fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 2 files changed, 11 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5ea1d64cb0b4..b20450dde5b7 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -652,6 +652,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "", #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif }; size_t i; diff --git a/include/linux/mm.h b/include/linux/mm.h index 53cfc104c0fb..f658923e76ad 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -224,11 +224,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -266,6 +268,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_MPX VM_NONE #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif From patchwork Thu Oct 11 15:15:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E785D112B for ; Thu, 11 Oct 2018 15:21:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5F302BA0B for ; Thu, 11 Oct 2018 15:21:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C8EE52BA15; Thu, 11 Oct 2018 15:21:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E7532BA0B for ; Thu, 11 Oct 2018 15:21:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8ECF56B000A; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A1426B026A; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 279AD6B000D; Thu, 11 Oct 2018 11:20:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id C4EF26B0269 for ; Thu, 11 Oct 2018 11:20:47 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id v88-v6so8157414pfk.19 for ; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7Y58Ryx2gRr8JA0dxpo0qOiFv/hhGNbXLRWK46z7PDc=; b=k80QAi5YcCXx05r7q2UKwwmgL8+Ur3OMrJ2qO8OFWJWpO+wsqZYndKQMR1iLjin3Ua nV//R7whL6qQ0MKHo6QJmeLpbVnwmQEWoDuvjL2X302mAuF4hH/UBQ0IQ0AztDmy7Ftm oml/Xbyj4m5wyv5cs/wg4iwedALeeiygiUXKr3+TeHHx5Lmak2ptXsexCNi/k7spD2Nv 11mC2HdHT2WfXYXy9NeZByjNpGp3XA2qbyqj+iFONlMSH87e8VGP1rFCvvRo+RedlGX6 KNWr2pCyzMuQtAkTiMkyi9/9lzp3k/IbmywTnJt+4HZQLaCdQzH9GGQvXzvENPkJ+yG1 PoTg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohu6OY8ZlZCStJ56qPWGfjPPDEIpXGpMF8KsvZHTX06rkS/dPgn PG4/yqNTAR+6AuSdJKXOgc+tUrx2SHAUZpVnxg0qlPHYgrai74OY1lAxCbXgpAzAIfPAmexHBFI uHzCSZiy0eX0xUnm9/o3KY+1RmN3u7pet/ENtlRC+zc0eX44NtYMUMHIT0ovPsZjtHw== X-Received: by 2002:a17:902:ab93:: with SMTP id f19-v6mr1971495plr.63.1539271247504; Thu, 11 Oct 2018 08:20:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV63tRkRP4Z0mwmYmwOLC1yom0LlWXxpPcIP3AnEP8hQOrWonUzLVzvufNLbjPgl63AVadfl6 X-Received: by 2002:a17:902:ab93:: with SMTP id f19-v6mr1971461plr.63.1539271246838; Thu, 11 Oct 2018 08:20:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271246; cv=none; d=google.com; s=arc-20160816; b=yr+xq2XFIoa47A2DAvdp50Z5Ghw7qCNrJKPNOXGosaEuQlkYkAbaoJBcDxeBiJJQYu SnrU4RYjGrEGp8DDau/Tv73jcTS9CBfWpXkJP8NXwfa0F2FZR5dMDsjCWPo1D+kzH46I dvD+DKZ2MOOz9839sqhTHovCTxLQAbfKIy2TLAsv5QYsNgp4A5PCzgDUm4rZRcr/DNwN n4d3sgvZwaCjxeIdZPMxQ+vYm/inhqhITcDKrDND/kb19fpuFUdBHBA2lgFETrp/CcBx ysg1FF8FHDlnho+UoZwt+HyNuqBBQ/J32APgNtyInS09Goohd5XAPBleS7h3AqA8zmWz 2AYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7Y58Ryx2gRr8JA0dxpo0qOiFv/hhGNbXLRWK46z7PDc=; b=vQrzD3jstK5Wk+SjnRoziPNy0j0YD5uodehcU/ryGiiggYkrLZXmmL9D285IcCXtE3 ByFWX3XLssbqNR679a1PFvmIiZkFgWleUceZE3pWnghnQ1lfSPoj3gupclSSWfoveE4I TWBz0ZsDXrACsgHW6ueIUn7BlnOCgJE8CgLss8xuBdUXKA4ct4pjbuIi4f6OCPuxQPiY oGWyjPeV9RAEXCrr0f76BejII86Cvan05XU05fo8TPbOZh8KDapytOP24ncBighY8UeF JDs5hhtGlfUIsz7XJxwmJbbQiJOVaLQoL2fyWwdSSSTfvdxaSqU0j/2MMW6c3E0uK8mf FxUg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id d35-v6si29570683pla.116.2018.10.11.08.20.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019110" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:44 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 10/27] mm/mmap: Prevent Shadow Stack VMA merges Date: Thu, 11 Oct 2018 08:15:06 -0700 Message-Id: <20181011151523.27101-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To prevent function call/return spills into the next shadow stack area, we do not merge shadow stack areas. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 5f2b2b184c60..b85292014ec9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1123,6 +1123,12 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + /* + * Do not merge shadow stack areas. + */ + if (vm_flags & VM_SHSTK) + return NULL; + if (prev) next = prev->vm_next; else From patchwork Thu Oct 11 15:15:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636879 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C45C112B for ; Thu, 11 Oct 2018 15:21:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 28EC32BA2E for ; Thu, 11 Oct 2018 15:21:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 268D22BA26; Thu, 11 Oct 2018 15:21:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78D472BA2F for ; Thu, 11 Oct 2018 15:21:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F3E06B027F; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 14D696B0276; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C43256B027F; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 616676B0276 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id 25-v6so4680202pfs.5 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=i21zJDW/boX++KSWpd1CddBLbgMI5qj32neFeniBxCQ=; b=KrR1w9ynyireEJkT8BO+PczxYQtWKtW+rzDZvaafe2GvAg4Glbc9IFe/syqhfucmPH NWTssFLZiX/D3buM3wnywoCdGH6rbv12MN2muto3I5FK6jSSMg1EcUFnVCWazpUWrhqL XID1iayFQ0NcgyLhP2byFoKme+BGN8tgw6hW5BTfA8tkiAIB4DHHyuAirXfJplASr7tp 0fA2UxCxkVR4RyhTZ8alxkC00MVs2U6rH8t3HNOL0+8rHloXl7a2bKRF/haWCjTb9Nto HtaxEh2l+M7dFHasK832wA27/IAqp1V6AN4MVGZ+GYS52bzmVa2oJ1D7THLXos30tRWW Li4Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogXOCMd/xxEC61X/uUI7MCEfdmqRC8a8qfHKWxm6x+3LwMnTgcu 5tRU4Bxj7NcoIjyu9O5vXRHWEzNgdPspeo8VyLE+6t4UrbRvfDJhpUngSaxg0NaRG/SOMqXQg6t GuCAx7EziPbG/TwG6P0kniLnHCWJmbbrt14gyjLBIEt7mcHP6QxmOQlcRoEwnzkEsXg== X-Received: by 2002:a63:db04:: with SMTP id e4-v6mr1837809pgg.280.1539271260063; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV61eM0hsuBEC4Fl6/jpAv0e6tXvSWtwJ0uTPBvSxWSHL6fQy+49Zr13C5Ce+zReypXjUg5cZ X-Received: by 2002:a63:db04:: with SMTP id e4-v6mr1837753pgg.280.1539271259139; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=dlcLuA0eZznDtCyhJixrlgNfXia+xPgy50ks7pMYm6NB9uyqyAt4Un85seoqGCzr7o vX9qlp+fopgNixsgm6wqTd+z4Xb9hLPPOhExCjc1gJvyo2+tw/sjcjWtiM1PoQkGbW2d CGihJC6uJu9l3RqlidCHEpeIvq72ZtA9HzYhpQlxFdeCtM1qawEv4Quq3JdcznE+l14Y VE6Sj1D15+RoCm+FzhNji32SHK7dKovRC3FJSYooVIpf6FYxgWWrJ4C0O+LIcMQYSba5 wqdPeFY0BZWjpzCAPfzSxyNaQOpIg6Rt2++OWOLiY1wf3txQphEL4mhyQw7Ud6BOWTVg erEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=i21zJDW/boX++KSWpd1CddBLbgMI5qj32neFeniBxCQ=; b=SMy7CQ4qiJMtFNOAp0lfm2soOx+KSFnoFgiGuhit0jlwztIs4/8NQ7Lokk2iFdIZTO BWB8Dn7RM5NstcYxgtIVSFRN67mwDJi/ZF75jp5svh84cFXE+EX4wKJzaLcm9HRPdsLF jYhyZY5PttvJ0R0hCiKbnnjY6v9KssF91910+m3yX16JmmUZ3GtVYwPOrVMSG8Ibxaim PPlGUP7Y+KkKrYaQ1LLniYqpzOInyJWdijN3KpKR9BXs859jmaoAWhvyZfA6np4LP+dp FWaC8Ha7cnojCMEiwxKsmHSAzxMjVnJ//K8Bf+eMGmN1ohX/gfviNaAbB8R1KFV41bm7 BPAw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019114" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:44 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 11/27] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Date: Thu, 11 Oct 2018 08:15:07 -0700 Message-Id: <20181011151523.27101-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We are going to create _PAGE_DIRTY_SW for non-hardware, memory management purposes. Rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 690c0307afed..95c918ad84ed 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -316,7 +316,7 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) @@ -390,7 +390,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -444,7 +444,7 @@ static inline pud_t pud_wrprotect(pud_t pud) static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index b64acb08a62b..0657a22d5216 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -15,7 +15,7 @@ #define _PAGE_BIT_PWT 3 /* page write through */ #define _PAGE_BIT_PCD 4 /* page cache disabled */ #define _PAGE_BIT_ACCESSED 5 /* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6 /* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6 /* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7 /* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7 /* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8 /* Global TLB entry PPro+ */ @@ -45,7 +45,7 @@ #define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT) #define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD) #define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW) #define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE) #define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL) #define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1) @@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3) #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED) #else #define _PAGE_KNL_ERRATUM_MASK 0 #endif @@ -112,9 +112,9 @@ #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) #define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * Set of bits not changed in pte_modify. The pte's @@ -123,7 +123,7 @@ * pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED) #define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL) #define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX) #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) @@ -187,7 +188,7 @@ enum page_cache_mode { #define _PAGE_ENC (_AT(pteval_t, sme_me_mask)) #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC) #define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER) #define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index 11eda21eb697..e7665a4767b3 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 612fd17be635..1b07080bd797 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -5833,7 +5833,7 @@ static int init_rmode_identity_map(struct kvm *kvm) /* Set up identity-mapping pagetable for EPT in real mode */ for (i = 0; i < PT32_ENT_PER_PAGE; i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r = kvm_write_guest_page(kvm, identity_map_pfn, &tmp, i * sizeof(tmp), sizeof(tmp)); if (r < 0) From patchwork Thu Oct 11 15:15:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5BAF112B for ; Thu, 11 Oct 2018 15:21:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3C492BA2C for ; Thu, 11 Oct 2018 15:21:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C17512BA2D; Thu, 11 Oct 2018 15:21:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B01C72B9F7 for ; Thu, 11 Oct 2018 15:21:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7EC36B0273; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A1E786B0282; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EE036B027E; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 155D96B0273 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id n23-v6so8045353pfk.23 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=XJFlpVxZM8SqCEUB+gtRXsphEAwyfwaAXYPRwoXrM3Y=; b=emtHGYKkgRm5TbL7nYiTARlZJK3QV9SWbwqXlqK379ElS5DErn6RQz/wdfGTInXgis L5ELODUmPPSJfL/EtTtIcQgUZo6aYe4fisVBj1BYlInnuspreUpE3A3YW5ZG4mIQ1KrI y26tzeka8Nnk+DD+AI1v4FKWSXFqwL6XDbnwGyrPFXyH5EyJ91gMvN3asAo9F5NZKrYU jZmdN0rNlHlU4A37LliD3YA7o0Li0Jh39X3N3OQb8Y4baOhSV8vmv60mkgXaKUoLTbyD aLgrgA2dRZ8+I2uFJgnTJ3ISCzLebneK2N5UtGsguZuMdAOfR0sfi1UJ7kw2ZyURcnzf oMXA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoi38ep2xVGDcEnV/yQD0UgLkd5jYhfUKZfd9U1UpPo8fSuXp2ix Qx6fQjhRM1NJ9/Jrs+BdyW2AKvoumVMiK2wZN8H40y/LJDU2EY7dVa1+oB6JVuxaT5N+yD6U7Pa r7jOKhkzDBqNHfjF9S1dZX20Li869rW55ua49CrJ3VQL72FM4BGicMYxrHXE3hB3JEA== X-Received: by 2002:a62:9015:: with SMTP id a21-v6mr2018170pfe.49.1539271259740; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) X-Google-Smtp-Source: ACcGV63TAzdbHrSofRh8uRMSlWmR1D7kyHlCfkCLeTuNv0hr9Hd71IuEWHe4K6QhucG6Dr24fLvG X-Received: by 2002:a62:9015:: with SMTP id a21-v6mr2018099pfe.49.1539271258594; Thu, 11 Oct 2018 08:20:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271258; cv=none; d=google.com; s=arc-20160816; b=UWU4RiOaoSGp0o7iM0Umwf+Athv2ID1StbkgK0SgImmK7JpwuGc9M1vkVAky3mR7GA k7dUzTdkBQm+5OKDKMZhzs09qu0xBSdP/VSzTS1gnfwrAO4mkwniTw5GEqaHB/ZJwcT+ /dQfodAuJOzgtSXp3u0oiZcg9KqZLk0IM15CJbgHN6F8MuApjADxuSXZXD5m09RjtVba OoBss+H0dlCMxEB2POJLNehn5BR8w1lLtXXLysjV+FNZEkswTXaRr+6TaRksBZirAq3u ITqDta5CHpj5ziGEZwvO+Mep7qFynpwGTKotJMTlVPxo6QCyY9lARcKbWOJPbi1ZH1Tl TodQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=XJFlpVxZM8SqCEUB+gtRXsphEAwyfwaAXYPRwoXrM3Y=; b=UmaHoyVw+R8pwDjM6tdZN5QgAwYanv15kq76LHekb+YD1ADk1yEoYtvhWRNOB99NOj 3elR5epqYWnd/+bNHrmlMdfJ2GiVbvfMtSx+DCHqbpVeXlJrVnC+LiqztRs54AtCwT9C yZNQNBfPZBgf0NIxE0MWbwHQekWOKCWxt4lY2F+6+VziAMCH9mSSR5KH/ccRxMYVvKMu lYd2Ia5h20/HZsilAo6F7iHB3p36IB5jsXmxZ0QmmBWvHzBJhbQDouZT8aUhmss6xgxV kXoWuRlAbhV7x3DZ9c513La1rnOJ1QkMujoN1CGXRiHbgQKsx3iBStNw9XeDpBh0YQNu rBZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:58 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019116" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 12/27] x86/mm: Introduce _PAGE_DIRTY_SW Date: Thu, 11 Oct 2018 08:15:08 -0700 Message-Id: <20181011151523.27101-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A RO and dirty PTE exists in the following cases: (a) A page is modified and then shared with a fork()'ed child; (b) A R/O page that has been COW'ed; (c) A SHSTK page. The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings: Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW) Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs. When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 21 ++++- 2 files changed, 128 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 95c918ad84ed..3ee554d81480 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -119,9 +119,9 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; } @@ -143,9 +143,9 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } static inline int pmd_young(pmd_t pmd) @@ -153,9 +153,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -294,9 +294,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + if (pte_flags(pte) & from) + pte = pte_set_flags(pte_clear_flags(pte, from), to); + return pte; +} +#else +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + return pte; +} +#endif + static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -306,6 +320,7 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pte_clear_flags(pte, _PAGE_RW); } @@ -316,9 +331,24 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { + pteval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + pte_write(pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + pte = pte_clear_flags(pte, _PAGE_DIRTY_SW); return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw(pte_t pte) +{ + return pte_flags(pte) & _PAGE_DIRTY_HW; +} +#endif + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -326,6 +356,7 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pte_set_flags(pte, _PAGE_RW); } @@ -373,6 +404,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + if (pmd_flags(pmd) & from) + pmd = pmd_set_flags(pmd_clear_flags(pmd, from), to); + return pmd; +} +#else +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + return pmd; +} +#endif + static inline pmd_t pmd_mkold(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_ACCESSED); @@ -380,19 +425,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pmd_clear_flags(pmd, _PAGE_RW); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { + pmdval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pmd_flags(pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY_SW); return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_DIRTY_HW; +} +#endif + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -410,6 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pmd_set_flags(pmd, _PAGE_RW); } @@ -427,6 +490,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + if (pud_flags(pud) & from) + pud = pud_set_flags(pud_clear_flags(pud, from), to); + return pud; +} +#else +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + return pud; +} +#endif + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); @@ -434,17 +511,22 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pud_clear_flags(pud, _PAGE_RW); } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); + pudval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pud_flags(pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -464,6 +546,7 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pud_set_flags(pud, _PAGE_RW); } @@ -595,19 +678,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); + if ((pte_write(pte) && !(pgprot_val(newprot) & _PAGE_RW))) + return pte_move_flags(__pte(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); return __pte(val); } -static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) -{ - pmdval_t val = pmd_val(pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; - val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); -} - /* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modify static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) @@ -1159,6 +1235,19 @@ static inline int pmd_write(pmd_t pmd) return pmd_flags(pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + pmdval_t val = pmd_val(pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); + if ((pmd_write(pmd) && !(pgprot_val(newprot) & _PAGE_RW))) + return pmd_move_flags(__pmd(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); + return __pmd(val); +} + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 0657a22d5216..bd17e38d6a36 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,7 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57 /* available for programmer */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -34,6 +35,7 @@ #define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ @@ -109,6 +111,21 @@ #define _PAGE_DEVMAP (_AT(pteval_t, 0)) #endif +/* + * _PAGE_DIRTY_HW: set by the processor when a page is written. + * _PAGE_DIRTY_SW: a spare bit tracking a written, but now R/O page. + * [R/W + _PAGE_DIRTY_HW] <-> [R/O + _PAGE_DIRTY_SW]. + * _PAGE_SOFT_DIRTY: a spare bit used to track written pages since a time point + * set by the system admin; see Documentation/admin-guide/mm/soft-dirty.rst. + */ +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ @@ -122,9 +139,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) /* From patchwork Thu Oct 11 15:15:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636871 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1D4B17E1 for ; Thu, 11 Oct 2018 15:21:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD16F2B9A6 for ; Thu, 11 Oct 2018 15:21:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0B462BA1C; Thu, 11 Oct 2018 15:21:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 40B3F2B9D8 for ; Thu, 11 Oct 2018 15:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 937D36B0271; Thu, 11 Oct 2018 11:20:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 873C46B0273; Thu, 11 Oct 2018 11:20:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C1F16B0274; Thu, 11 Oct 2018 11:20:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 25D956B0271 for ; Thu, 11 Oct 2018 11:20:59 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id o3-v6so6467656pll.7 for ; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=p+UvYcP9CZ4wyf+seELTfYJWQiVrathxQP30tHpCFzxvV5OLJU/6V8CPC79FwhqcPc wkNpjoeZ8MUNHDGuBLKCQUVvaxdDJZn8Cp87iZtHU4MiO0Jmq7GaGEXXyuBYIvu51rR2 4oAqBBTrgX/qQdcaXIjRQM4GLbWDaQC8fd4Yfue7lP7rc/MqKla+ctZBkBvxiM4oIw9T ovmwGoJpu5ein+BFliJPBPKPfspEntDb69g2DHtSefetiNbA04XwuzSRivJ5qIyBboVp EdwpjJSl94J9vlAMY4KIlnpdZJmjLkhemz+A8uJBnM8cWHZ+YbQ4yw2FXCdxF1LZqSyO rDHA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojquDqEqE4OEDZhR2SB2dx4864Nx+GkweMweTSv9oc09ipL5q4/ yaI1iXH+pfjcRCtaj1aVNI98CO8t2jIUkd4LZBdrkhEroFg6ReAQzbduKf1kV/jU83T0AJie9NY EnAkZeuL7hFf6tSHBddEIFrC3WepsFp0WYCWKxdiO2wMUmY083YCxWGq7eyLzcdJjGQ== X-Received: by 2002:a63:3203:: with SMTP id y3-v6mr1833871pgy.101.1539271258848; Thu, 11 Oct 2018 08:20:58 -0700 (PDT) X-Google-Smtp-Source: ACcGV63OYOgh32+LhTgII9cHSD7N+pxedtPLJnV8YvN6SveiXLPEZ626J5gbZverLdCdogYTZtKo X-Received: by 2002:a63:3203:: with SMTP id y3-v6mr1833840pgy.101.1539271258297; Thu, 11 Oct 2018 08:20:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271258; cv=none; d=google.com; s=arc-20160816; b=sHsFaSgpREC3sryUETipxGtyoLvax9fTj5FkZpJTPi8qDpOQPUhTYkNMapK1ooK5rI SBqZuxSG9647GOSYTLilQuRzBNavKZKN59GlYMGOJacj/HHoO2ZOXWb2bYvQioylmYfC lFAumelO9V2shwQUWmvGWCYcOK7GpZ4G+tC0hceto8uasrcdwadxGrUiJD5CHV5hF8FQ yjQXwp079vphxoFsLLXcTffWPJ7+QgmesvUo9/mEVAGpgkdcVbPnMtzsCMc0FaTbBQNE Sc62TeiOK7MBBWqIyjqFrmOxSY/e9Xw/gVOAi16t4d4Z9BevYGNDDqMI4cdK3tzXPOPD vxdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=hQuOiNGr9KRbeU0+fQuv+wQrjfgIYsowVrlixZJZ7WB+3RyaoDbBiVMZGrh36KVP6/ 1FYD6j4onlGTlT/4JfkQvdlZlukZ2Jg6+eUSgtoul6VYVFgOQX+hjKklot0f+kLUlKw2 fuZ3cQX/PXKKxqm5K7ihGP0LTHxfg9o9uUw7l20o8kMYlzf8zSsUAC6j+7vg9MedAU3X FK1ONWscxwyGcqkAEraFvxF8OdnFH43VJXs7lSYe1woDVMsuQ5YIQN1Gxi7YfGSHJysi 70f/VtyvcdG3fwwh4SDpBI4xT6O9/JD2mIS/+TUrhiYfvobAtGsk4HTo+6BN65Fj1fln Dhdg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:58 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019119" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 13/27] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Date: Thu, 11 Oct 2018 08:15:09 -0700 Message-Id: <20181011151523.27101-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Update _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry(). In order to support Control Flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW. Signed-off-by: Yu-cheng Yu --- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 00aad8164dec..2d6ba1462dd8 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1170,7 +1170,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, } /* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse(se); ops->clear_ips(se); From patchwork Thu Oct 11 15:15:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636881 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E03917E1 for ; Thu, 11 Oct 2018 15:21:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BB1D2B9F7 for ; Thu, 11 Oct 2018 15:21:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2F4E32BA29; Thu, 11 Oct 2018 15:21:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95C052BA26 for ; Thu, 11 Oct 2018 15:21:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E1BC6B0287; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4EC596B0289; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9CD76B0280; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 88C766B0280 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id e3-v6so6548374pld.13 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=fbIU10g+3FIH2F9V39N52dFDZKOYUCa0+VKpaAcOC2w=; b=AI4Ajq8szF974oPQ/R7lDnagYpy/lWuBzdgT4+0Y40W6ZEqfzd0VQ17ThRUNyVMWeP XY09ColEv5njC0raFL/FFg3rnvunfVtBIUB4qQe0G1MQRcwBTzm3oZTvm+pVRY1QU/4B Zi1WZhWeaRQiFnrIgFENZ7P84yRvbouwRpbAIXdAhKyc/0zJDXjzO4Snw6PVoE6vDpVr SyqNYCRdXdt7eiVSSLFFNRezXBTSBTZ/iIZRBpRa6m3sw9WqIkT+bih6sBTolYJfGAOZ Z0z3VQPIFPp7AJ/lPbt+oS8qKCc1gcPjqW9XIa8GMrAg5Na4btUsXCRfVM5wf4uUn8TE yBDQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojWTEUtZRH6OOrjGxuOWO3QEnaKJ3/53YDz6O/rpud5bpz1olHI rGARwynW49m7qlWMttHY8ROXG5mu3Dx5Eyj/NLaOQgLpWI1V4LTwuZDZfHXbafsHYjFboOb2L/x 7yrEwz6mXlzJhGOsQbpLO02c3SQr6nIw0ZOLqpmgj83l0Z7usZb71HlJtAYNmHosuTg== X-Received: by 2002:a62:3a84:: with SMTP id v4-v6mr2005771pfj.118.1539271260247; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV61xDSYN8RhJnqfMqD0kvPwhOBw0hlQQHZcJqeJKkHd9v/iKZbu3aIV/kiE1r11Ccv1vQBSM X-Received: by 2002:a62:3a84:: with SMTP id v4-v6mr2005710pfj.118.1539271259341; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=xRRcwYC727BRhmegGYyf+Ah48qia8TGo2p997StfGmriSTCivnA9kwiDfS+cxnKWW9 ux8FVT4Y7mLmDEp+9SCNAm5Y9+cxl3oAl+T13N5CGvlhwm0XHczlmbFZZWYynzQ0/I4S qIqc6/m5CtdTzsaDdHERcWyAevckQd1t60pEp2n3pEtnnRLVxdHVWB7nsDpGDr7qRHsf e4uBJLvGva5AXi3qp2kZcf2Ec1W+vF6T0UtPhXb9pOsMUiewVJk/a0gNn5H0cH6UiMTt kS6xmDdC4kO+TtiERcERV4Pxwr8Ey6zG2NejZYZXNBqgrRKyuIU59cIB0GBDpXs4IZSY pSIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=fbIU10g+3FIH2F9V39N52dFDZKOYUCa0+VKpaAcOC2w=; b=wL5PN9OL/48e9qvEuSjuj6sxHGK6KeitWc5jh2OywrXlYhbvJIvhR36xFEifZZIGhS 9OyH9OquzG3Y6LOrxJPNTeN2SJvm79x2/o9E/X4d0TSLQUh+zNL05Uxw4vBMMq0VaBI5 Rrsa/WSVZqU3oZ5cGo+BBvopdaN1FV9+vi6M5YlYgFzsKkRmfvUkPfMqnbiC/O5YWXOr 8BIL1E40eMhUEweFiJNvgx4csJYFZW2/GnUyNyhvqxHx3Oya5AOSY7N8Ddh70/CnQgb3 36dyU1ua1QcaJAGag8IJ62uKICdfq0bvz15vuAo59I1uaDC9YhGnfCxRwY1toVQnTbjm XCsg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019123" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 14/27] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Date: Thu, 11 Oct 2018 08:15:10 -0700 Message-Id: <20181011151523.27101-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When Shadow Stack is enabled, the [R/O + PAGE_DIRTY_HW] setting is reserved only for the Shadow Stack. For non-Shadow Stack R/O PTEs, we use [R/O + PAGE_DIRTY_SW]. When a PTE goes from [R/W + PAGE_DIRTY_HW] to [R/O + PAGE_DIRTY_SW], it could become a transient Shadow Stack PTE in two cases. The first case is that some processors can start a write but end up seeing a read-only PTE by the time they get to the Dirty bit, creating a transient Shadow Stack PTE. However, this will not occur on processors supporting Shadow Stack therefore we don't need a TLB flush here. The second case is that when the software, without atomic, tests & replaces PAGE_DIRTY_HW with PAGE_DIRTY_SW, a transient Shadow Stack PTE can exist. This is prevented with cmpxchg. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 58 ++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 3ee554d81480..b6e0ee5c5503 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1203,7 +1203,36 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pte_t new_pte, pte = READ_ONCE(*ptep); + + /* + * Some processors can start a write, but end up + * seeing a read-only PTE by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PTE and flush the + * TLB would be needed. + * + * When changing a writable PTE to read-only and + * if the PTE has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PTE is + * not a valid Shadow Stack PTE. + */ + do { + new_pte = pte_wrprotect(pte); + new_pte.pte |= (new_pte.pte & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pte.pte &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(ptep, &pte, new_pte)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +#endif } #define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) @@ -1266,7 +1295,36 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pmd_t new_pmd, pmd = READ_ONCE(*pmdp); + + /* + * Some processors can start a write, but end up + * seeing a read-only PMD by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PMD which looks like a Shadow Stack PMD. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PMD and flush the + * TLB would be needed. + * + * When changing a writable PMD to read-only and + * if the PMD has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PMD is + * not a valid Shadow Stack PMD. + */ + do { + new_pmd = pmd_wrprotect(pmd); + new_pmd.pmd |= (new_pmd.pmd & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pmd.pmd &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(pmdp, &pmd, new_pmd)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); +#endif } #define pud_write pud_write From patchwork Thu Oct 11 15:15:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636873 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AAC6112B for ; Thu, 11 Oct 2018 15:21:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17B3B2BA19 for ; Thu, 11 Oct 2018 15:21:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B9032BA23; Thu, 11 Oct 2018 15:21:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BDE3A2BA1A for ; Thu, 11 Oct 2018 15:21:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F18E6B0281; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5AF956B027F; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42F586B0276; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id EB1336B0272 for ; Thu, 11 Oct 2018 11:20:59 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id l1-v6so8094644pfb.7 for ; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=k4I+IBksg1YUf85BSJ9dR9AdYeEvkDMb/4N/mPBj2zE=; b=HRlN/KJdQZcGXtKCx2XXppjfMmuG/W1g/R1wqGqFJbHJV2e/sba2QRGt2aKvEZFE/c Jar1eaKWtSTtEpKob4UaI4xzFLYClTQO9gpLOriy2YIbvh1UBrKGxdKiusWifQH1hk1L tfNF/LMBitdU2b7I/nE1KlwAQr3fLCw+WPK5ZRlvu00OFJkRxr1uhPlVjPkVNJQwkPRu Kqxo8xGg66ykR2bzp0OMclZkS15FeYECeKDE8hPLWze7j1OssFQpdJ9RX25XKTMoiCmO KzvxURwzMhaHSEDFRP5W+yzk1WWbhoZe124POWpfsKSJ/eimW3tO8jgFKjhiVqCie78k fj+g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohse09Jy289G0pZxQRGwyFupS3ls3ctGanfTv946E9a7J4+iLYu cc/JcDh5uNCfrQAA3KOp2uFyBMbxSwtp3KSlWVYNoHeFf1VPEKAu0TpUHknOaLHhVtv33s1+oKw piFly45WBzGpUVWzB2s1ou/5AOMy81Puz2f3OxE9a75/1tumtQ0ywDAV1gpVmUjnzSA== X-Received: by 2002:a63:e05:: with SMTP id d5-v6mr1827752pgl.272.1539271259626; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) X-Google-Smtp-Source: ACcGV604hSovoGjLczemV6i0bJNhS5ECDI18z63iVWQ2mGyrygyNtZTETh8CrItMrZtn1giW52VC X-Received: by 2002:a63:e05:: with SMTP id d5-v6mr1827716pgl.272.1539271258859; Thu, 11 Oct 2018 08:20:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271258; cv=none; d=google.com; s=arc-20160816; b=ItShhpQ4RkxHTlB5UNGyLLC9b3CCrd/VV1LoxTjx5AYCMoCHmspBlLoivus9J5at4G gSBTvxwoyo075PpM2enO/GBoyzz49zmEad38arKtG5gwsAn02wzl86xliyQvV5R8IOMB iDP1TgAKdbmUIwmsL50gQ+rTdU847ORSmgpXta8+1oSv0qjds7BXho1lnEms9N0cCxK3 SarD9lPu13rIy97EQcHadGtFZJ6h0jrEGVIqynP/Xpmc67F7Ky4NwBAKbYhG61Aa4ZMs jCyVfyMfJTweOqnMG+nFk0WBHp2g5vAF+jHpp2VSMvmwIIhLJIjf/XfUqq3HT5N2KjYv Skzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=k4I+IBksg1YUf85BSJ9dR9AdYeEvkDMb/4N/mPBj2zE=; b=dIkdDtCvwC+USjuz5S/PCH42fIpuHxhJXJ0XOtJH4wXUii7KJdyIAi2M2xA9ZorOhU f2x33BIBQ0iu7fJHqKhQkI6VPYOqkKUm7JuoITOdcdRz6yS/MDLQKevRTWU5xeYB0Uo2 lODVDFiB3PiAz8iO1r4ch2dQbhity9yPVJ48vCth1vPxJB0kMZIzoVWQEuh6Kev5hiGO /QMQCVvNq0Pcx0B+BLI3HKVzsZ7djGcEJap9R0PIjK5OLkdSWJMoglgf7wK+kgNQMVkR 7kFEJ2vXTBXVAHufvjIVZPPjDi5RVfR5Z2s4JgzTJ3JXh0WbzkqaIn0nYU0GO+yeaodO a9lA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:58 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019126" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 15/27] x86/mm: Shadow stack page fault error checking Date: Thu, 11 Oct 2018 08:15:11 -0700 Message-Id: <20181011151523.27101-16-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set. In access_error(), we check if a shadow stack page fault is within a shadow stack memory area. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 5196050ff3d5..58ea2f5722e9 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -157,6 +157,7 @@ enum { * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code { X86_PF_PROT = 1 << 0, @@ -165,5 +166,6 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, }; #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 47bebfe6efa7..7c3877a982f4 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1162,6 +1162,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Verify X86_PF_SHSTK is within a shadow stack VMA. + * It is always an error if there is a shadow stack + * fault outside a shadow stack VMA. + */ + if (error_code & X86_PF_SHSTK) { + if (!(vma->vm_flags & VM_SHSTK)) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ if (unlikely(!(vma->vm_flags & VM_WRITE))) @@ -1300,6 +1311,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * If the fault is caused by a shadow stack access, + * i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set + * FAULT_FLAG_WRITE to effect copy-on-write. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR) From patchwork Thu Oct 11 15:15:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636885 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D22A112B for ; Thu, 11 Oct 2018 15:22:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F01DD2BA2E for ; Thu, 11 Oct 2018 15:22:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EDB6A2BA3D; Thu, 11 Oct 2018 15:22:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 523B72BA34 for ; Thu, 11 Oct 2018 15:22:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA0166B0280; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A95626B028C; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D7DB6B0285; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id E82936B0287 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id d63-v6so6451010pld.18 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7rUGwSivKJuLXOaScdHGzjdpvton/iET1Ot+BK5m7Jk=; b=V26CE5OW6uHPgSZb0zjGZunbqoqE5MOzkd96C1MlEBb31NFOi9D+Ql8KkyzAzFztK1 08julxF6PVsrHWKhwRIg6lEFcZruYtPPC15shVooMZ6aKfgUCv9xARTLZB7gcu76ed0n l0YQ/lG7DuJX+pChyfgjgfk8B4g/vhofrWnDIey9d37Rkt5YGrBKCQqphDWJg2Ci4CJu 4Hk+YAHpG8Hx4Xd6r2oCYhXj/TzwW+eRpS77zgTUGyUnghAOtHMiArujqESmDIZHhOWI U48/5g8O9004KxOqEazoQL56j0ThVccez6Sguwv/v0pi55Cqy6Zm+gqlri6JETHYQGOJ IDDw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoile8A26fp/b6XI8Tvnl2Zf7RmWmo5mx3HCMpTbDKoPIc7ohSZA biFalPObtEoJSBjmv+rZcVSj/F+rr8ixbj2fzmbAX95r8KsHtYsWIa0nXY2P8yiy/cy2nSyo3ku qH9uhbw+ug4qnnDOYPMgPMM9ONcof0sEaqiLku3wZwemzdXp+zYDvDFXdW5J2jnTlUA== X-Received: by 2002:a17:902:e28a:: with SMTP id cf10-v6mr2027255plb.81.1539271260611; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV62GagMgTH57ZAJFLzcNP/WM/8DTQ2ieH/l/r+mmy6kjHTZFbNezI7r01ZDRe2dtvqq0a2Hi X-Received: by 2002:a17:902:e28a:: with SMTP id cf10-v6mr2027202plb.81.1539271259616; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=CtTV4cHh3dxKoqaNdRCbkkuHupugQDYtbY525Ph7bGvlCDbl7zMVfcVzz4F63QNJjy ckGh7/lgMz8PQGbkTVKI7r9KgyM/y1ZeDjxkUJ+mfMqGiRIAwBHZNcU4Msd6UATKvXw+ 9H3cpyMeu/ileQ5MSGJ5eDrJ/beWuopMjyP2Wpf4FIpdGXHzxcfukvIreXpE0SAii81b WlposWMhKCM9Ne+F4PN6BmxgQWddc8+z921J82Dc/oqbK9bGiVQhtI9Aund79gi6REdO 8k57dI9LiOZ4abo3kxhKfuo4mBjCdENZw07StPB+gAtVQsp0RV9BrDY/+hG4DdUfmvgn Dpwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7rUGwSivKJuLXOaScdHGzjdpvton/iET1Ot+BK5m7Jk=; b=CEhkGPgtC3Q2cmeheRjawUnnRfeDBNt2hKcYn3AE0GCIjGGTblfVBMcxQulmee071+ yvK8FYh1UFpx07dxQzx4vlKUCV605NVT1isjENa7zYqYgbRt6CpLGB7GK7Wq9eNlzws4 RxG+QBXp8h/GSSicw7Jqpms+j58+WK38cChttgg5A8pF0L3wTGT27CU5yhFrXvEJD8JI ffkO1+YIifM+sXzV+oZ/HsZB7w0MaedTQ5lPD7UyfH3a2NghNXa2pMe3ZS/8pMzjL8Oa 6PPcrLJT5pQQIo4b3NZDcOQ+EG3smnPZ2Y+WSM/osnpVZxXfTrZCs6NBxG7NwI0YLUue /RmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019130" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 16/27] mm: Handle shadow stack page fault Date: Thu, 11 Oct 2018 08:15:12 -0700 Message-Id: <20181011151523.27101-17-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a task does fork(), its shadow stack must be duplicated for the child. However, the child may not actually use all pages of of the copied shadow stack. This patch implements a flow that is similar to copy-on-write of an anonymous page, but for shadow stack memory. A shadow stack PTE needs to be RO and dirty. We use this dirty bit requirement to effect the copying of shadow stack pages. In copy_one_pte(), we clear the dirty bit from the shadow stack PTE. On the next shadow stack access to the PTE, a page fault occurs. At that time, we then copy/re-use the page and fix the PTE. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 15 +++++++++++++++ include/asm-generic/pgtable.h | 8 ++++++++ mm/memory.c | 7 ++++++- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 089e78c4effd..e9ee4c86a477 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -881,3 +881,18 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_mkdirty_shstk(pte); + else + return pte; +} + +inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) +{ + return (vm_flags & VM_SHSTK); +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 88ebc6102c7c..015b769377a3 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1127,4 +1127,12 @@ static inline bool arch_has_pfn_modify_check(void) #endif #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +#define pte_set_vma_features(pte, vma) pte +#define arch_copy_pte_mapping(vma_flags) false +#else +pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +bool arch_copy_pte_mapping(vm_flags_t vm_flags); +#endif + #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..1fb676ec7da2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1022,7 +1022,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * If it's a COW mapping, write protect it both * in the parent and the child */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { + if ((is_cow_mapping(vm_flags) && pte_write(pte)) || + arch_copy_pte_mapping(vm_flags)) { ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -2462,6 +2463,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2535,6 +2537,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -3045,6 +3048,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte = mk_pte(page, vma->vm_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); + pte = pte_set_vma_features(pte, vma); vmf->flags &= ~FAULT_FLAG_WRITE; ret |= VM_FAULT_WRITE; exclusive = RMAP_EXCLUSIVE; @@ -3187,6 +3191,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_set_vma_features(entry, vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); From patchwork Thu Oct 11 15:15:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636887 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0BE5112B for ; Thu, 11 Oct 2018 15:22:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C97412B9DB for ; Thu, 11 Oct 2018 15:22:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCFC02BA3A; Thu, 11 Oct 2018 15:22:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2540D2B9DB for ; Thu, 11 Oct 2018 15:22:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010D46B028B; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E8B866B0290; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B30426B0289; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 57D3D6B0280 for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id b7-v6so6250989pgt.10 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ICzAJYaQm7Rxm6UBhR7njAsEqpmzxdHr9fB/FjNVRU8=; b=Bkeq0pEWOEoiWdN6RiZ9QZ1ADBDPvTQ1gYWt0DcOpenqUxY2TWXowcYaOhhCko+2ZM 6IQ6NFWAbAYKDg9EKoq7kHMvTaT/Wg45KUI7/VL6Fur7bRCRWxnhIvxEDhvSBFZeF7lH 0yUiqcJOj2mILjZUDgcS8GKzVzcAeKQZpGoUouNNyUMYkgVQiO6T/RR0tdVVWach8wFC /+IQWYz56krSTpmDRQ+E9SxtLcntD3pJlTzOUfL8Bu96JyopbD7mhtUSetvXW4DcDoc3 sxF3Kb2srh1y4gjA8eOeWsa7aS/T8NkpTf6FBuFGArUx7pr5SEoHDaJHGJWBjEqUXj0j 0DBQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogiAmSNwn3TPgSiYcfQjixqSvQyeM1tRld/OQobq3BLEgiQJAAT dLQ4h6s/aq7n6HzPD9H2Byqsj8gJXuvjWJZESALt3914q+URk8HpzNm1rZcEyLa1P9JRpl/21ce +LlSg3ztkqH50wQfm/fjIY+0frBmpdFSKVzLEepAm9ct0m2xbeqd3Tjfzo6xLm6JQRA== X-Received: by 2002:a17:902:70c3:: with SMTP id l3-v6mr2061259plt.125.1539271261048; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-Smtp-Source: ACcGV61MRRwI3PrYORqZe2DdMdb7GgIKsBdtGmFbTLPvwh9Cpvt/yEv9r+xm9ibl0KBLrvYp2hXg X-Received: by 2002:a17:902:70c3:: with SMTP id l3-v6mr2061208plt.125.1539271260249; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=UecrZpIF9YN9xAUK1pq9JESlEV6KkST40b4Kx1gBS6lwgTcqej2lS5GxYmpprMATTH hnow/fLKKS1UU348/rO3/7DK2WIGlt4J8TjEbXLVM0dY8/8bfO3wMWhzhMKghVJjbxU8 ud2t0l+VXMSV712nkaG8OzxsYqb+4RRD7e5cbvgaOFr3snrLVPvbFnArXroMkfMKiFOm RUE9TIYkKb4igkjY1Mss9ODq5EPTLZ4eT4YOOVdSB851Td5k1bP1Q3Rxp5A/GXeCVbvC Il8RY8kvo3Mc0BC4UESnlCJn7SLvOPNJQw/Z0coc2QmB0p0xENMSTUBurdreuRls8QUC ugZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ICzAJYaQm7Rxm6UBhR7njAsEqpmzxdHr9fB/FjNVRU8=; b=udg5cr85bVz4ixj7z239B4mptlbzf4rTIIS6zAgQ4fNOEBSYD/hrlULW0aU1Ru9VOZ 6EEKnzxcZEUPounhlUTetBK+RW00aJQOiB5MTXGhseC7fsVibXTYiDT/6u0YpLyP7MH7 u3g1lTOkHFMskWdhiksH1p9jFL/l/QP3pNyi7snC+jWS7iz0v6bjkbh/Sk3BcbUAVtXh g7nnmq8OInQlaGUvp34vmv2vSJD1ZVhXRin+iHtHabWG8gPVLfeqlgX3BQYYoGbedzdB BQ1GBdCoJzPg4ucB4pzpNVqyXVSF2qvh26mO1gHUFlKoAIx1k550yI8K0900y+aug+H3 xMXw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019134" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:45 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 17/27] mm: Handle THP/HugeTLB shadow stack page fault Date: Thu, 11 Oct 2018 08:15:13 -0700 Message-Id: <20181011151523.27101-18-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch implements THP shadow stack memory copying in the same way as the previous patch for regular PTE. In copy_huge_pmd(), we clear the dirty bit from the PMD. On the next shadow stack access to the PMD, a page fault occurs. At that time, the page is copied/re-used and the PMD is fixed. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 2 ++ mm/huge_memory.c | 4 ++++ 3 files changed, 14 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index e9ee4c86a477..864954bda7fe 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -891,6 +891,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 015b769377a3..7512e4dfd642 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1129,9 +1129,11 @@ static inline bool arch_has_pfn_modify_check(void) #ifndef CONFIG_ARCH_HAS_SHSTK #define pte_set_vma_features(pte, vma) pte +#define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 00704060b7f7..6e03e26c1cec 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -597,6 +597,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1194,6 +1195,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1278,6 +1280,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1349,6 +1352,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); From patchwork Thu Oct 11 15:15:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636889 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28F6817E1 for ; Thu, 11 Oct 2018 15:22:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 175FD2BA40 for ; Thu, 11 Oct 2018 15:22:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B6CA2BA36; Thu, 11 Oct 2018 15:22:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 653242BA36 for ; Thu, 11 Oct 2018 15:22:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45C7F6B0276; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 283016B0292; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00D746B0289; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 315996B0282 for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id r81-v6so8066856pfk.11 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+iDfpNVUh/DPLyWC38GpRC9D68g0kKsuLFXmOgWEuXA=; b=oxZEuyuBHQK65JnP6TYzLI9DPmaIqp9Lr/+uW3ay+uGIGLvJCrdO04Px9bX/1llJ+b 289yWSQa+rdNLNH4vubSbzLghwXKHPnbQEYLGVBODB0gexFAzdUUOV+Yi835X/RqcXlf NwXGAzL99mhscJ64aoxKXzuXlR707VWGs8EemmCOOOivqi8MJaBaNXJBjCwbRZQhNBTN iPmJSnEmvgHExXjOTL+FehSxPAdQESFbMntG4V8L9c+Ve2mWPQi7f2BuN6l60vFzRDVO 4TY/hBdpYrp+/DRNn8+LJC8x1F/Nih9ImPfC+GFlCwQfURS3CSQ6TrostAMyRuTTJsv3 ZF9w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoiFy0rnQCo1Q6ab3iZYRP5CI3WpeJMlJLLPB8WRiqtasf0ip75K C9OFZT7D9Q3Id+hdS2WL4QW961pXc17aojIq/ZqiRhxEd8hfogtC4rtyVw90KlFJoCSlR2cD4XI FliZXgu8+iB8uPx990WU2E8lLxGVItuqwVZtN/RpSvUElFSYmR2T6cNm4NBvyVMgsfA== X-Received: by 2002:a17:902:b70d:: with SMTP id d13-v6mr2010010pls.44.1539271260877; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV622pSV+KEfCF2q/KTiEI2h/wHzpNmuA3p5iJVApNdJdF+It7DWOXHo++SoPwFQ+zI0pE70v X-Received: by 2002:a17:902:b70d:: with SMTP id d13-v6mr2009974pls.44.1539271260049; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=hX8/N2Jgz8I2ztfv7Y0RZA1eyui0K5A4PkxE9e7vl2oIJBHTNDAZkFxzWWdz3UeKvC IxBAXO+S+APoxeedxvo2Dlyk2lG8KLPHvURjJS84zZ4NU7/ooEDveYXRcivDsTitGqM3 mqWMDzf2uCnnKVwRGX0BsezHQspQ4t2v94WBxYc+mx7KQTPFdSVApOfxTL0/rjvtmyKo P8Vd8Ge5ThHfiGUUS18c2Kl4IsXM65IEnIsdwIQh7Jixc47m0i1Sir9xAiSTyVSNr0gO vGxuS3nzg0OUf5X5pBE69Xwu/MlxQDLH+cN08M/SzPkhwI/p3BV78c/b9qgGgKh3Z9W4 Aosg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+iDfpNVUh/DPLyWC38GpRC9D68g0kKsuLFXmOgWEuXA=; b=PZ7CAsFI+7IgsBJia4vpn4lRzBSfwnveRqLSUVTir+zW6eVA+TXGgFUJbCtYlsmfQr R/DPuyK37h70wth4mWMb4ACtoaxalkpjz0OAb5CHHB+OKYDM0yx2KALlaI763sd11mtv 4Y8CV+7xHYSTTUse0XcJI7Df7U+ITJaNdJ4rnvhld48wnX7hoD+z+lBAB+SoIS9SHt/T cZeNMQm7lZVKPheaoHQZbn1hMFnK0pkbO3V8bR/z7Uiho7hfeaKHsACeRnGQZHBLvHg1 6gTfhfBeQrk1eQD/AcnMVlIwBybUoPXmyzCNKV5WbrCWDcN0/0CAgMtY3XwtKZgeA8Je bG7Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019139" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:46 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 18/27] mm: Update can_follow_write_pte/pmd for shadow stack Date: Thu, 11 Oct 2018 08:15:14 -0700 Message-Id: <20181011151523.27101-19-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP can_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW. A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW. Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive. Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?" Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 18 ++++++++++++++++++ include/asm-generic/pgtable.h | 4 ++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 864954bda7fe..80876b2d03b7 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -903,4 +903,22 @@ inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); } + +inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_dirty_hw(pte); + else + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_dirty_hw(pmd); + else + return pmd_dirty(pmd); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 7512e4dfd642..09881698a566 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1131,10 +1131,14 @@ static inline bool arch_has_pfn_modify_check(void) #define pte_set_vma_features(pte, vma) pte #define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false +#define pte_exclusive(pte, vma) pte_dirty(pte) +#define pmd_exclusive(pmd, vma) pmd_dirty(pmd) #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); +bool pte_exclusive(pte_t pte, struct vm_area_struct *vma); +bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/gup.c b/mm/gup.c index 1abc8b4afff6..03cb2e331f80 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -64,10 +64,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte(pte_t pte, unsigned int flags) +static inline bool can_follow_write(pte_t pte, unsigned int flags, + struct vm_area_struct *vma) { return pte_write(pte) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pte_exclusive(pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma, @@ -105,7 +107,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; - if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) { + if ((flags & FOLL_WRITE) && !can_follow_write(pte, flags, vma)) { pte_unmap_unlock(ptep, ptl); return NULL; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6e03e26c1cec..3b3e1026fb5b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1387,10 +1387,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) * FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags) +static inline bool can_follow_write(pmd_t pmd, unsigned int flags, + struct vm_area_struct *vma) { return pmd_write(pmd) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pmd_exclusive(pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, @@ -1403,7 +1405,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, assert_spin_locked(pmd_lockptr(mm, pmd)); - if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags)) + if (flags & FOLL_WRITE && !can_follow_write(*pmd, flags, vma)) goto out; /* Avoid dumping huge zero page */ From patchwork Thu Oct 11 15:15:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636877 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CD69112B for ; Thu, 11 Oct 2018 15:21:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0BD342BA1E for ; Thu, 11 Oct 2018 15:21:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F2E3A2BA29; Thu, 11 Oct 2018 15:21:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97B4C2BA22 for ; Thu, 11 Oct 2018 15:21:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9C976B0274; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF7396B027E; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A6886B0283; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 35AC26B0274 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id i81-v6so8148012pfj.1 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0PZbYToGLHKhMY6f2I+7PcGrhLgjeHE3A2EtpGQ9/js=; b=tHT53h2OMyY/HCkS2eFfDYGtmFWdHEObIR43mJPEa6IBQLpjaNBREN/3UeqXJZZ1tf BgbrcEykCSavZtQBeGZD5ssUKwZcuxw75XPX12wPNm2fxJ5poOib16/K1gksjxcTRA3E 5HsAScocvxKHUGQ8dfkPvhbVqhRDb+UvMTttO5dFF+Tc/G7GI2iL8n0ptBQFKtYWLVpE SHKJqBFXn5giIPlttG1zfJwYoV4NlncyA01BtxwUWd0GL+SStuhe9GOAs0/dJYc1ZRvq yG1GlD5MX2GOq3QscvwApf4WMdM3rgOylGFHbfTMLJP4MGBoVv2cB8ZI3Tj3Yh4CNAUi hzVA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoi/FxF2kvfc9US4TG0y/HlZxMn2DVDjzgFkCxxKRBiSOQhNmGCF BjmLGLlu3cOlzEgFmIs5i5rLl17cQttu9nNYakkdKsFEREaPJZoDLaO/W+e9br4TkqSOcANbDGn mq5x546ijmV25ONl0rNBtQNPlnaIJY5QAwklfxzG1vdta8nuJ3DYuwlA8junHt1iJfA== X-Received: by 2002:a17:902:7408:: with SMTP id g8-v6mr1924038pll.221.1539271259917; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) X-Google-Smtp-Source: ACcGV63JkPcacGjwtb5YAQz22+liCAjce0/mcX+GxKQTO3HznXB6tlxgqYEkv07R+oNx8e0VoULY X-Received: by 2002:a17:902:7408:: with SMTP id g8-v6mr1924004pll.221.1539271259372; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=wY5qfOXBA8EZcT4sb+UsFDkAIiyJuBk2+xlAnZxp7uAqCJtZYXs3UtS/lUuiLPwmfU z2uLmpj2x3suJj3osFJb2BSydY+kiA8OZY4R9X6rf7G7jseagikxGLOL7NOvcvjIScEc kufnkB15DZI2xUKcOVXS3WNPskDMjF8T2tuurAfpzmFCZ9QXFLQR8YyjvTCNBNh7PyMD 0CMkcNxyiN2LOXQVkcmhXokiJhJK9bD7pFaATkQsK92qMQFlqzUMPIVOgzvuVmxFv/7u 1YVFp3KwWM4DexMsv3RoezPDPsSVtLE1FTmE4OdNh5+wHuaehpAdc8zjujfgj9orioPF tQkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0PZbYToGLHKhMY6f2I+7PcGrhLgjeHE3A2EtpGQ9/js=; b=zPonTE2AODJg7ONfatGau8XRdD+xy3GacyeO9uAa2OQ/QP/j3aPRSKtJs/IuDnOMXS TlGN4EM8Yx+S4e9ez+gFIAIfPuv7+u5FaZonQ8FOBuuhWx2xZ0WvlB9DTluFhCiq1Bx+ CDDRBf0jdDVfJogo7BMBZ8xAy8x6hSIc4RlZyPgwFEqBsw5gwu47bDfnbcx9ZgXbqTeF jZAcVq4uwurJYUl+92neokA6s2MgeAnuuY2av7tfx1zrEjphQ/7NvUHuGyCoh8Tt2WSg 0KhAYae5Wqcnfzj8kRLkGOnThVLG2ex3XWHCRoheqS0kmohj3/k1VYkmD+xSs1+auAR3 Rpww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019146" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:46 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 19/27] mm: Introduce do_mmap_locked() Date: Thu, 11 Oct 2018 08:15:15 -0700 Message-Id: <20181011151523.27101-20-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f658923e76ad..de55f367852e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2318,6 +2318,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len) static inline void mm_populate(unsigned long addr, unsigned long len) {} #endif +static inline unsigned long do_mmap_locked(unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags) +{ + struct mm_struct *mm = current->mm; + unsigned long populate; + + down_write(&mm->mmap_sem); + addr = do_mmap(NULL, addr, len, prot, flags, vm_flags, 0, + &populate, NULL); + up_write(&mm->mmap_sem); + + if (populate) + mm_populate(addr, populate); + + return addr; +} + /* These take the mm semaphore themselves */ extern int __must_check vm_brk(unsigned long, unsigned long); extern int __must_check vm_brk_flags(unsigned long, unsigned long, unsigned long); From patchwork Thu Oct 11 15:15:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2739917E1 for ; Thu, 11 Oct 2018 15:22:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1264A2BA4C for ; Thu, 11 Oct 2018 15:22:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 061032B9F7; Thu, 11 Oct 2018 15:22:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 167BE2BA3C for ; Thu, 11 Oct 2018 15:22:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69D906B0291; Thu, 11 Oct 2018 11:21:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 625FD6B0296; Thu, 11 Oct 2018 11:21:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C9B16B0297; Thu, 11 Oct 2018 11:21:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id F36076B0291 for ; Thu, 11 Oct 2018 11:21:09 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id j9-v6so2615579plt.3 for ; Thu, 11 Oct 2018 08:21:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=tq1L2b3oU7If8baeoND4gL66llD8CMcxxdVLNm/Slyg=; b=ub3b+tp5kP3OO1SVFZzDCa613tn+T62JOToioP9UhyQTLl37YYljR2oZvhhPVu7cFb Vd2BreGJH41iDJHzLyF0ytOXsGqi9zsUGzFgHuVotsfRmeq44gjrbVJVyttxWRjxn6Z1 ZEODZUxFJvzU/uaMqx9bRslPtb290+RFQXrKzwQSOrEYnLrFbnj9yEiqL3ncKhq31UVE 5TeiQ29ABi3FtV5PAde1BY2FoS36sGRvE9vCK4V4VsDBGx8NU4WiGrLNSwNqQ0/Yl/vJ 0HuMBohzkRLZTJBOkX4ZxR7+957qt1OLFp0ocFFvx1jjXmpF2oE5Kov/mWjqSLr6Wh2B 7PXA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoibkpKjOsav+ZDsrogQFdjJtUXa+ph2KRW8Jf/k8uH/lbonrkUM 2abFSzX6RHV+I+jPVfCagsqRk9LRvpG8WB8chbL2byLdGALFmjpstEyajzaltVb24x/TaNvaxOa i16TvVbnPL+eMBkpTWtQ2yqTvMkMErY1MQ0FZ9hnx52UHPzgGifSp+kXqPeIGvsOv2Q== X-Received: by 2002:a17:902:7c8a:: with SMTP id y10-v6mr1949500pll.322.1539271269638; Thu, 11 Oct 2018 08:21:09 -0700 (PDT) X-Google-Smtp-Source: ACcGV63qvARGLiybAUoFaGIK2Jr1TIIltJEGvb5AJ6EcLyZt+PszVS+Qmi7KJarTgRXzz8k3U7kK X-Received: by 2002:a17:902:7c8a:: with SMTP id y10-v6mr1948966pll.322.1539271260022; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=xmaUCFB7zvKEzuiBxIhu/DMmDWkdrQlbxIfiFUA/842cvOIplYT0l/44o7cPvvcoP4 WiYYVzw5Gq+kglSazM5BdL2EwAYtNp+KW2Em7NsbIuXi8FSlcauskQ+A6RaH0gmRhXs7 qVSCAjEqUkLzZrLEODdnae9iKM8+m1lCYqusNUw7iA2kCZxerypEvPxyMSAY2fLbxlso peoVbhp9RxkaaYxYfmP/mGONglfTWsCt0Yy5aSQXWugUYYB6ZN8DhejRKVUO43OTZVcc U+ZaDksgzE1N/6WFqodOYdkivp58o24B+zE/78/oteQp8M9B3+YDebWmxu2ftsudDF/h s+vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=tq1L2b3oU7If8baeoND4gL66llD8CMcxxdVLNm/Slyg=; b=ulbNO2+0Por64dtmZHgR9aF/mRQCcIEB9aVODaZ7ImIwEWpG5At+gE+hXlX0MHDKt9 Oew0VGIakF22CtSNv4ExmUUQc/AidJbjuFBtp/BjpSed2Y9QNJX3LEhQkKPBg5NgAKwG /o5aVQZdFmlNNdhpR0yeJHIkfluZaYUSF4/Ea/kSCtQL0YjX0hBzGFel3J93udRWEExp t4Vo2FnXjLdmGT4yf71YLA03jPyjPounDbokRs3tN8RlMByGpo7raHldWDDg+ruV7SiL kmENf1lZxwpO2nrcNc3KqPjRKig9meGKyWVSbL+OZVe2Gh1FwLXc+btiSDEF5n4pDcTW LSnA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019157" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:46 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 20/27] x86/cet/shstk: User-mode shadow stack support Date: Thu, 11 Oct 2018 08:15:16 -0700 Message-Id: <20181011151523.27101-21-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. The shadow stack is allocated to a fixed size of RLIMIT_STACK. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 34 +++++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 14 +++ arch/x86/include/asm/processor.h | 5 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 109 +++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 25 ++++++ arch/x86/kernel/process.c | 2 + 8 files changed, 198 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h new file mode 100644 index 000000000000..c952a2ec65fe --- /dev/null +++ b/arch/x86/include/asm/cet.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; +/* + * Per-thread CET status + */ +struct cet_status { + unsigned long shstk_base; + unsigned long shstk_size; + unsigned int shstk_enabled:1; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk(void); +void cet_disable_shstk(void); +void cet_disable_free_shstk(struct task_struct *p); +#else +static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline void cet_disable_shstk(void) {} +static inline void cet_disable_free_shstk(struct task_struct *p) {} +#endif + +#define cpu_x86_cet_enabled() \ + (cpu_feature_enabled(X86_FEATURE_SHSTK) || \ + cpu_feature_enabled(X86_FEATURE_IBT)) + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 33833d1909af..3624a11e5ba6 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -75,7 +81,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 4731f0cf97c5..e073801a44e0 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -777,4 +777,18 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0 /* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2 /* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4 /* kernel shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7 /* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8 /* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index d53c54b842da..63918cecf367 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -24,6 +24,7 @@ struct vm86; #include #include #include +#include #include #include @@ -505,6 +506,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET + struct cet_status cet; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8824d01c0c35..fbb2d91fb756 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..ec256ae27a31 --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,109 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * cet.c - Control Flow Enforcement (CET) + * + * Copyright (c) 2018, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int set_shstk_ptr(unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + return 0; +} + +static unsigned long get_shstk_addr(void) +{ + unsigned long ptr; + + if (!current->thread.cet.shstk_enabled) + return 0; + + rdmsrl(MSR_IA32_PL3_SSP, ptr); + return ptr; +} + +int cet_setup_shstk(void) +{ + unsigned long addr, size; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EOPNOTSUPP; + + size = rlimit(RLIMIT_STACK); + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + /* + * Return actual error from do_mmap(). + */ + if (addr >= TASK_SIZE_MAX) + return addr; + + set_shstk_ptr(addr + size - sizeof(u64)); + current->thread.cet.shstk_base = addr; + current->thread.cet.shstk_size = size; + current->thread.cet.shstk_enabled = 1; + return 0; +} + +void cet_disable_shstk(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return; + + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_SHSTK_EN); + wrmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, 0); + current->thread.cet.shstk_enabled = 0; +} + +void cet_disable_free_shstk(struct task_struct *tsk) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || + !tsk->thread.cet.shstk_enabled) + return; + + if (tsk == current) + cet_disable_shstk(); + + /* + * Free only when tsk is current or shares mm + * with current but has its own shstk. + */ + if (tsk->mm && (tsk->mm == current->mm) && + (tsk->thread.cet.shstk_base)) { + vm_munmap(tsk->thread.cet.shstk_base, + tsk->thread.cet.shstk_size); + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + } + + tsk->thread.cet.shstk_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 44c4ef3d989b..c3960326b67f 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -52,6 +52,7 @@ #include #include #include +#include #ifdef CONFIG_X86_LOCAL_APIC #include @@ -411,6 +412,29 @@ static __init int setup_disable_pku(char *arg) __setup("nopku", setup_disable_pku); #endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ + if (cpu_x86_cet_enabled()) + cr4_set_bits(X86_CR4_CET); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk(char *s) +{ + /* require an exact match without trailing characters */ + if (s[0] != '\0') + return 0; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_SHSTK); + pr_info("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n"); + return 1; +} +__setup("no_cet_shstk", setup_disable_shstk); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization @@ -1376,6 +1400,7 @@ static void identify_cpu(struct cpuinfo_x86 *c) x86_init_rdrand(c); x86_init_cache_qos(c); setup_pku(c); + setup_cet(c); /* * Clear/Set all flags overridden by options, need do it diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c93fcfdf1673..4a776da4c28c 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -39,6 +39,7 @@ #include #include #include +#include /* * per-CPU TSS segments. Threads are completely 'soft' on Linux, @@ -134,6 +135,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(tsk); memset(tsk->thread.tls_array, 0, sizeof(tsk->thread.tls_array)); + cet_disable_shstk(); fpu__clear(&tsk->thread.fpu); } From patchwork Thu Oct 11 15:15:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636883 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 207B017E1 for ; Thu, 11 Oct 2018 15:22:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EB0C2B9F9 for ; Thu, 11 Oct 2018 15:22:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 02D612BA2D; Thu, 11 Oct 2018 15:22:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DBE12BA2A for ; Thu, 11 Oct 2018 15:22:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92E426B0283; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 824516B028B; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 454706B0283; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id DE7266B0285 for ; Thu, 11 Oct 2018 11:21:00 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id j9-v6so2615062plt.3 for ; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+LUlDEYrptrR0R8FWohMCAMq0vezAJBgWs/8dNmgsIE=; b=JTcFv5OBHio0jg4YH2/JGPv/1fbCBhKHpndPhfH2izYqN2EdhANJNQ3Jv/6tGoJfbZ fQdkhfZIQiEjg/KoXCt1V+cPU7g3KI0kC/O56md84TSOR1QKp+g00Goi0d3GbS7YmSBp uqnRTLdGWYE9FcN8peLj183/wyddeCZP38hPQNCLrvyHK7l1vWhgU/lxwggM7k6hnapo gBZFdjIaxEa5F9b4kZyOUTf++l7lqipPzIPOA5W9nxKFS6ROQMoj2n3wZ+oUHy5VkA7v KxMJKLEg+TIXJ8/kLfpMeK/3Z/5EheLfrm+1q3qV2/k5a79fx6KDObQ7c3HfAcMHxExA jLWw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoitiAhnC+3TtKy3VukjpzWHCrrTcqdfvcZEYKjHAmwnAuyXnihd yVNxUAVJbHtMKSZUtw4p/kKXp4+7gf+6d6McAg7E7p7LsaMSJz8njzGt1IP8qyxIr0oFXUeq8Oz yijanRPGiDpgZbnhhlQNrMR2SUqVTBFgSeyagcitQVpiBD5qBDRvD33NzsxZc4j2t+Q== X-Received: by 2002:a63:6c84:: with SMTP id h126-v6mr1821755pgc.237.1539271260573; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV63fLNKhUkp5rJ6y62YJ/MXflSv7jbtCbAGfFazy26nH7MnYsEscSt8zeiPsqdSRzeTTe2XW X-Received: by 2002:a63:6c84:: with SMTP id h126-v6mr1821714pgc.237.1539271259823; Thu, 11 Oct 2018 08:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271259; cv=none; d=google.com; s=arc-20160816; b=BU9GIUn3e/2oYKcvbAV1iwGmfSJ4vkfrITlJZZ+Gj41mzJEJ1Rfua12hLJQ3HsWGAU rrp5xuYppNKxTDcPHdsh1PneJp3ZNyEubqtZbriQRKADqv+brJHgbx1hwg8LT/m2el4J o9kLKeM6vvMj/krvy1RDniL2FhUpZdDf9wLl2AaxCVRrOduKH6nyPAY+Q26gqIyFHD6M OmA4RTa2W3Mm8FemGwiCaZcJTx4rmod51CenFblN595yozamiBjb0rmDB8Tcqcm7dFuy puuge62TvGO2czd+lAdMmoKMGvmNwk2R7RqKjJyGR2KAGa2KTqLuaoZTzhu6DpXgXthF CBiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+LUlDEYrptrR0R8FWohMCAMq0vezAJBgWs/8dNmgsIE=; b=VUJGDr0S6+jeX6ffSm3+g7E7k/zqwJy+GE2Mq/z+y2opACwTTELjuW/hSXso2WY4yb HpngCyseu+UTkastEZoklBPUa4T8+7IDsq4EXPF31lUBOn4/TjV8AVrg8id1lzBusyuE PiX3t3MDy9+ydUbiV+xblyWvJI9GTbFztBK79sEnRxfIEUnMy5os9hW69pKJ61FAzlJl sKOfTdmxPc8GmR/4yO2iC9ma3hGKGydvR55ORB1IsvW+LEv1+rF2EXFy56Vh2KlBe3GE VL8Ntax1WyJBoP0TVnFZLmBr2VnFMrnUneK0EgBhVwiU7tpPgq+CQV+cLeC20oH/7Jq/ +QJg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.20.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019158" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:46 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 21/27] x86/cet/shstk: Introduce WRUSS instruction Date: Thu, 11 Oct 2018 08:15:17 -0700 Message-Id: <20181011151523.27101-22-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler. This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does a fixup. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 9 ++++++++ 2 files changed, 41 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 317fc59b512c..37f16269747d 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -237,6 +237,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + asm_volatile_goto("1: wrussd %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + WARN_ONCE(1, "%s used but not supported.\n", __func__); + return -EFAULT; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{ + asm_volatile_goto("1: wrussq %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#endif /* CONFIG_X86_INTEL_CET */ + #define nop() asm volatile ("nop") diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7c3877a982f4..b91fc008f33a 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1305,6 +1305,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, error_code |= X86_PF_USER; flags |= FAULT_FLAG_USER; } else { + /* + * WRUSS is a kernel instruction and but writes + * to user shadow stack. When a fault occurs, + * both X86_PF_USER and X86_PF_SHSTK are set. + * Clear X86_PF_USER here. + */ + if ((error_code & (X86_PF_USER | X86_PF_SHSTK)) == + (X86_PF_USER | X86_PF_SHSTK)) + error_code &= ~X86_PF_USER; if (regs->flags & X86_EFLAGS_IF) local_irq_enable(); } From patchwork Thu Oct 11 15:15:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636895 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A36EF17E1 for ; Thu, 11 Oct 2018 15:22:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9111C2B9A5 for ; Thu, 11 Oct 2018 15:22:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8EFCA2BA48; Thu, 11 Oct 2018 15:22:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85B332BA21 for ; Thu, 11 Oct 2018 15:22:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09BC06B0289; Thu, 11 Oct 2018 11:21:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BA3586B0292; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FA386B0289; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id ED4706B0282 for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id t9-v6so6506139plq.15 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=bBl/34N8RFFR+BCwegEHDoncf10kbC/jokQM9Co0DDY=; b=JA/bsRBPjgHo4Erm+dXDpRqBs/Wryg4MBVZUsAwXvmuwmY/B/vGlL18u5INB9SqEeb XmJilagqrY5cFaPUIiJxj4+OUCBnLA1UhxN07jpn6qRpjP2CKxkQwq6Qj5bwBdwa712n tMWllGp7AtidZAczcJfbbjsXq+kCV0pqjIFHoPz5fExgKJMgxv57FRIM0pqinmmtglYj ohbTOTAjNCDMBTsXkkCitgbGWZR8M4lskyX44tyBYiEM33qtkgM7zzBeCBdGYpYVP7ZF A8u1pGtKxqOWonY5SYpd/Q5WiWOezPm0in1UFU0VwbhKZ4/F4nQ3dSvA1zsCJOHy20Lu y6pg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohb8AXDApB0OBZTt9i3Kau1u/A9PlEmJG165Q9uO1YFVNPZJ9vz K+Sj6n1OoBpd9E+NY9ifCYxy/x7AuESjOU8x0l8/kvx58cuBN5VGk8ZRx546vKmcY4m3sU+AuBK a4HT9z1ooO2gzlkJE1j6k0ZO3ASIHxtHczpgSYL3CxaEXtt0CH4Z4DLyO+ZAux1gIlg== X-Received: by 2002:a17:902:447:: with SMTP id 65-v6mr1956898ple.325.1539271261613; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-Smtp-Source: ACcGV610v1rGeSS7vn6U4aYbTiPpaQnciym64EUErYokTBXO6fWlcNPDMc5JtM6+g8yDE+ZlzQ/Z X-Received: by 2002:a17:902:447:: with SMTP id 65-v6mr1956831ple.325.1539271260290; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=Em5F5JGn2QppqmTYCmnoJR1l+YWnzzsgDhXrEIliQQvK+szCcyo6Wp3A6/xW9G/kZI LY1sLpjUPQ0vzPPBGtSQA/0p5RyD2UfcXXxXnQeH4Xd9g1gHaWe/C18ggUfcq/uKgUjH K82btzYFuoiEWdKPwv+pQ6eDRNXp0Js27rls3uXk3kVHXw/2pNtN1Tq+dYcwfw1J/S58 JoBAZI5SlDTWcMF0wBlwoZQEOE8QGnOmba/r+89c5WhH+4oqPHyF215LEIZKq+qYIemh POSNFrqMbjsXOzh582WnY+BEmaiK3VpMuaKp2kuSs+jhe7N3O05YWXLmiBTQHCWREl4k 2rjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=bBl/34N8RFFR+BCwegEHDoncf10kbC/jokQM9Co0DDY=; b=hd1rZIh9RdrYRjBqL8j5BXk0fzDM3bjMzD0SXxh47Q2I+DEj4RTMH5Ef3wEPWiwN/O zeO0UKweBuu+KK/UDJ9NCo+MmSKpkzOYXuGTswElS7DuSE2MHVEDDPCFdosCk8zvwMC0 x4W7ue2WY0qLJIrZe6CVukNbf2VHELHKYEVtcIRDlsoym2LUu2qv3HTt+kTE8RHJo5ry 3x3Lt+lL4ayYgVsswY8WuXJ5EHc36w/SRuj/pUeaT45UZ7QqLVkuCbZa6n9Of8K+qxTI WMr9vwCF1J4j8gFFNqyU2BwB+JyXVMde5nl04FuszPudNehg1aZSYIbHMvGP6HWRTcZQ y/Xw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019161" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 22/27] x86/cet/shstk: Signal handling for shadow stack Date: Thu, 11 Oct 2018 08:15:18 -0700 Message-Id: <20181011151523.27101-23-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer. Signed-off-by: Yu-cheng Yu --- arch/x86/ia32/ia32_signal.c | 13 +++ arch/x86/include/asm/cet.h | 5 + arch/x86/include/asm/sighandling.h | 5 + arch/x86/include/uapi/asm/sigcontext.h | 15 +++ arch/x86/kernel/cet.c | 126 +++++++++++++++++++++++++ arch/x86/kernel/signal.c | 97 +++++++++++++++++++ 6 files changed, 261 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 86b1341cba9a..cea28d2a946e 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -34,6 +34,7 @@ #include #include #include +#include /* * Do a signal return; undo the signal stack. @@ -108,6 +109,9 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, 1); + if (!err) + err = restore_sigcontext_ext(buf); + force_iret(); return err; @@ -234,6 +238,10 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, if (fpu->initialized) { unsigned long fx_aligned, math_size; + /* sigcontext extension */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, @@ -277,6 +285,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, if (ia32_setup_sigcontext(&frame->sc, fpstate, regs, set->sig[0])) return -EFAULT; + if (setup_sigcontext_ext(ksig, fpstate)) + return -EFAULT; if (_COMPAT_NSIG_WORDS > 1) { if (__copy_to_user(frame->extramask, &set->sig[1], @@ -384,6 +394,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index c952a2ec65fe..3af544aed800 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -19,10 +19,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); +int cet_restore_signal(unsigned long ssp); +int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} +static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + unsigned long *new_ssp) { return -EINVAL; } #endif #define cpu_x86_cet_enabled() \ diff --git a/arch/x86/include/asm/sighandling.h b/arch/x86/include/asm/sighandling.h index bd26834724e5..23014b4082de 100644 --- a/arch/x86/include/asm/sighandling.h +++ b/arch/x86/include/asm/sighandling.h @@ -17,4 +17,9 @@ void signal_fault(struct pt_regs *regs, void __user *frame, char *where); int setup_sigcontext(struct sigcontext __user *sc, void __user *fpstate, struct pt_regs *regs, unsigned long mask); +#ifdef CONFIG_X86_64 +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fpu); +int restore_sigcontext_ext(void __user *fpu); +#endif + #endif /* _ASM_X86_SIGHANDLING_H */ diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/uapi/asm/sigcontext.h index 844d60eb1882..e3b08d1c0d3b 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,21 @@ struct _xstate { /* New processor state extensions go here: */ }; +/* + * Sigcontext extension (struct sc_ext) is located after + * sigcontext->fpstate. Because currently only the shadow + * stack pointer is saved there and the shadow stack depends + * on XSAVES, we can find sc_ext from sigcontext->fpstate. + * + * The 64-bit fpstate has a size of fpu_user_xstate_size, plus + * FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext + * is located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; +}; + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index ec256ae27a31..2000e5e87f8f 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -18,6 +18,7 @@ #include #include #include +#include static int set_shstk_ptr(unsigned long addr) { @@ -46,6 +47,80 @@ static unsigned long get_shstk_addr(void) return ptr; } +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64(token) ((token & TOKEN_MODE_MASK) == TOKEN_MODE_64) +#define IS_TOKEN_32(token) ((token & TOKEN_MODE_MASK) == 0) + +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +static int verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp = 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (ia32 && !IS_TOKEN_32(token)) + return -EINVAL; + else if (!IS_TOKEN_64(token)) + return -EINVAL; + + token &= ~TOKEN_MODE_MASK; + + /* + * Restore address properly aligned? + */ + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + /* + * Token was placed properly? + */ + if ((ALIGN_DOWN(token, 8) - 8) != ssp) + return -EINVAL; + + *new_ssp = token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp = 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr = ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |= TOKEN_MODE_64; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -107,3 +182,54 @@ void cet_disable_free_shstk(struct task_struct *tsk) tsk->thread.cet.shstk_enabled = 0; } + +int cet_restore_signal(unsigned long ssp) +{ + unsigned long new_ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + err = verify_rstor_token(in_ia32_syscall(), ssp, &new_ssp); + + if (err) + return err; + + return set_shstk_ptr(new_ssp); +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, + unsigned long *new_ssp) +{ + unsigned long ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + ssp = get_shstk_addr(); + err = create_rstor_token(ia32, ssp, new_ssp); + + if (err) + return err; + + if (ia32) { + ssp = *new_ssp - sizeof(u32); + err = write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp = *new_ssp - sizeof(u64); + err = write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + set_shstk_ptr(ssp); + return 0; +} diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 92a3b312a53c..a0cee6ba99a0 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include +#include #define COPY(x) do { \ get_user_ex(regs->x, &sc->x); \ @@ -152,6 +153,10 @@ static int restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, IS_ENABLED(CONFIG_X86_32)); +#ifdef CONFIG_X86_64 + err |= restore_sigcontext_ext(buf); +#endif + force_iret(); return err; @@ -237,6 +242,17 @@ static unsigned long align_sigframe(unsigned long sp) return sp; } +static unsigned long alloc_sigcontext_ext(unsigned long sp) +{ + /* + * sigcontext_ext is at: fpu + fpu_user_xstate_size + + * FP_XSTATE_MAGIC2_SIZE, then aligned to 8. + */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + return sp; +} + static void __user * get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, void __user **fpstate) @@ -266,6 +282,7 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, } if (fpu->initialized) { + sp = alloc_sigcontext_ext(sp); sp = fpu__alloc_mathframe(sp, IS_ENABLED(CONFIG_X86_32), &buf_fx, &math_size); *fpstate = (void __user *)sp; @@ -493,6 +510,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig, err |= setup_sigcontext(&frame->uc.uc_mcontext, fp, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fp); + if (err) return -EFAULT; @@ -576,6 +596,9 @@ static int x32_setup_rt_frame(struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; @@ -707,6 +730,80 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) } } +#ifdef CONFIG_X86_64 +static int copy_ext_from_user(struct sc_ext *ext, void __user *fpu) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_from_user(ext, p, sizeof(*ext))) + return -EFAULT; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + return 0; +} + +static int copy_ext_to_user(void __user *fpu, struct sc_ext *ext) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_to_user(p, ext, sizeof(*ext))) + return -EFAULT; + + return 0; +} + +int restore_sigcontext_ext(void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + + err = copy_ext_from_user(&ext, fp); + + if (!err) + err = cet_restore_signal(ext.ssp); + } + + return err; +} + +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + unsigned long rstor; + + rstor = (unsigned long)ksig->ka.sa.sa_restorer; + err = cet_setup_signal(is_ia32_frame(ksig), rstor, &ext.ssp); + if (!err) { + ext.total_size = sizeof(ext); + err = copy_ext_to_user(fp, &ext); + } + } + + return err; +} +#endif + static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { From patchwork Thu Oct 11 15:15:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636931 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9C35112B for ; Thu, 11 Oct 2018 15:23:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 934891FE84 for ; Thu, 11 Oct 2018 15:23:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8518E2917E; Thu, 11 Oct 2018 15:23:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D20A1FE84 for ; Thu, 11 Oct 2018 15:23:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D78976B02AA; Thu, 11 Oct 2018 11:22:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2B446B02B0; Thu, 11 Oct 2018 11:22:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCEC76B02B1; Thu, 11 Oct 2018 11:22:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 6A93A6B02AA for ; Thu, 11 Oct 2018 11:22:54 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id d69-v6so6188674pgc.22 for ; Thu, 11 Oct 2018 08:22:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=3eT3Cdn0PbjC3gGYHqwCGZejUd2BcmT8ZYC6emS5Dc4=; b=VNeSF6pKf89i5d/NjZXGNKf2NFfDfVu9FKmxl4lRscweXDrOkdJMVeArfJnnIPCsN1 HI0rO8SdNL4tqGbRPlazmjvkKx2KnRMA+mSs8e3pR3j4VsGw1+PfrTDwlV3GX79DkfOy Jrii1qZvJClyFuxi/3lCI2btUv+8krODatpwJp+KaIsx6Iwlej0FDxBjOcHhQISpwQ68 W16t0fa8vhnUo+hAiLlzyXUUFYywDpRkdXPelRPcr9t4SRswMMxD4DVu4sXH8DR5ZYPE sZh4weQ95ox76Hf5lcaO7hwBRvaDSj6p2TM8NguqQ1Es6kb3E3zClVGjKyZH5XNi9jQG UotA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojnfjQSnQIYz/3hxgO9cpo+a20YxgtVljr0aFqVJQqMLu/eefbp uj0OhFnDJn4LhRptFiqI5nDh6r6I9Y6uk3WweoUwazEJpV9G+MYIUQ+nAk9HAyKsBSwG9YbvI4y YR3D3LugQXi20HfNJOuUnj+S+CxEKxp9gZwbObEwsWd4Oi/EdZzRKCuucdrntXvCzGA== X-Received: by 2002:a17:902:7109:: with SMTP id a9-v6mr1998079pll.310.1539271374018; Thu, 11 Oct 2018 08:22:54 -0700 (PDT) X-Google-Smtp-Source: ACcGV62Sq67oPvgQOA5pmHxcpeUcGCufwhn+T07RTYmq4fBvFpwoN4m/bP+UbpHxOU+mMCoSOeIj X-Received: by 2002:a17:902:7109:: with SMTP id a9-v6mr1991768pll.310.1539271260470; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=ha8QJPerIpCu8s2C6Ft7nC8F7/TE9DcRb34NRYQhHd2P1uq1+RchRI3da1oQQCy/ar DYy/sVZ2f7u7bag8TAIvJ6as0D3RxZi6iY94rpyYz0nYfrtQe07IvrTKEredFaRgeLQ9 6P41LKt1LgUC+RYRTcLm5jxhHiul5O3K8ghMP/khsQ0Nr0YcO1gOdLMVLKpqhhKloS6v pKVVmuebgRHTS+ytXG36P7M15mlrWjwDErQu6FfF6g8GGz3y7m8mLE68UBheBoob7PUA exln+TjgamW7oLZQkIo9Mu7xMs8QLGC3eZc10LO+JzcCMcLbWPwN0sqKm6Zhd3QUH5d0 xEDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=3eT3Cdn0PbjC3gGYHqwCGZejUd2BcmT8ZYC6emS5Dc4=; b=I52Wewn1hygPkwlWuXMulLaLrOQmNKYiwZy9xNdyfp1cqL7F3MQC9AqlU+soPQgZDM teLGr1AADCdfzkPpUYNU+UplSix4E/jKS4uOvpLPH9gHRwl8maQN1Z+/y8OStNzpwmmk TFY4ht55iyijeuE5cwsd9A8QbgnJg4n/e/ObOihRZpKTVKfvLyWx9G8572hYf4cfdgoT Y0ZeFzmq2Eabar29FVamBmZ9GgRSk7Yv7+s2mr9wM0WSmqUdnua4VAXy8LX+FTEHSgbQ sqm6QB0iS+VDQGQ/uCpsilA1RU2C8+wQ4j6Kj9kyWiGyBR6x7kG4g0J4eundLnOuoTlP La/Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019166" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 23/27] x86/cet/shstk: ELF header parsing of Shadow Stack Date: Thu, 11 Oct 2018 08:15:19 -0700 Message-Id: <20181011151523.27101-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 4 + arch/x86/include/asm/elf.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/elf.c | 341 +++++++++++++++++++++++ fs/binfmt_elf.c | 15 + include/uapi/linux/elf.h | 1 + 7 files changed, 383 insertions(+) create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/elf.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5f73335b7a3a..ac2244896a18 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1919,6 +1919,9 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_PROGRAM_PROPERTIES + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n @@ -1927,6 +1930,7 @@ config X86_INTEL_SHADOW_STACK_USER select X86_INTEL_CET select ARCH_HAS_SHSTK select VM_AREA_GUARD + select ARCH_HAS_PROGRAM_PROPERTIES ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..5b5f169c5c07 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -382,4 +382,9 @@ struct va_alignment { extern struct va_alignment va_align; extern unsigned long align_vdso_addr(unsigned long); + +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +extern int arch_setup_features(void *ehdr, void *phdr, struct file *file, + bool interp); +#endif #endif /* _ASM_X86_ELF_H */ diff --git a/arch/x86/include/uapi/asm/elf_property.h b/arch/x86/include/uapi/asm/elf_property.h new file mode 100644 index 000000000000..af361207718c --- /dev/null +++ b/arch/x86/include/uapi/asm/elf_property.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _UAPI_ASM_X86_ELF_PROPERTY_H +#define _UAPI_ASM_X86_ELF_PROPERTY_H + +/* + * pr_type + */ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* + * Bits for GNU_PROPERTY_X86_FEATURE_1_AND + */ +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + +#endif /* _UAPI_ASM_X86_ELF_PROPERTY_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index fbb2d91fb756..36b14ef410c8 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/elf.c b/arch/x86/kernel/elf.c new file mode 100644 index 000000000000..2e2030a0462b --- /dev/null +++ b/arch/x86/kernel/elf.c @@ -0,0 +1,341 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Look at an ELF file's .note.gnu.property and determine if the file + * supports shadow stack and/or indirect branch tracking. + * The path from the ELF header to the note section is the following: + * elfhdr->elf_phdr->elf_note->property[]. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The .note.gnu.property layout: + * + * struct elf_note { + * u32 n_namesz; --> sizeof(n_name[]); always (4) + * u32 n_ndescsz;--> sizeof(property[]) + * u32 n_type; --> always NT_GNU_PROPERTY_TYPE_0 + * }; + * char n_name[4]; --> always 'GNU\0' + * + * struct { + * struct property_x86 { + * u32 pr_type; + * u32 pr_datasz; + * }; + * u8 pr_data[pr_datasz]; + * }[]; + */ + +#define BUF_SIZE (PAGE_SIZE / 4) + +struct property_x86 { + u32 pr_type; + u32 pr_datasz; +}; + +typedef bool (test_fn)(void *buf, u32 *arg); +typedef void *(next_fn)(void *buf, u32 *arg); + +static inline bool test_note_type_0(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + + return ((n->n_namesz == 4) && (memcmp(n + 1, "GNU", 4) == 0) && + (n->n_type == NT_GNU_PROPERTY_TYPE_0)); +} + +static inline void *next_note(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + u32 align = *arg; + int size; + + size = round_up(sizeof(*n) + n->n_namesz, align); + size = round_up(size + n->n_descsz, align); + + if (buf + size < buf) + return NULL; + else + return (buf + size); +} + +static inline bool test_property_x86(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if (pr->pr_type > max_type) + *arg = pr->pr_type; + + return (pr->pr_type == GNU_PROPERTY_X86_FEATURE_1_AND); +} + +static inline void *next_property(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if ((buf + sizeof(*pr) + pr->pr_datasz < buf) || + (pr->pr_type > GNU_PROPERTY_X86_FEATURE_1_AND) || + (pr->pr_type > max_type)) + return NULL; + else + return (buf + sizeof(*pr) + pr->pr_datasz); +} + +/* + * Scan 'buf' for a pattern; return true if found. + * *pos is the distance from the beginning of buf to where + * the searched item or the next item is located. + */ +static int scan(u8 *buf, u32 buf_size, int item_size, + test_fn test, next_fn next, u32 *arg, u32 *pos) +{ + int found = 0; + u8 *p, *max; + + max = buf + buf_size; + if (max < buf) + return 0; + + p = buf; + + while ((p + item_size < max) && (p + item_size > buf)) { + if (test(p, arg)) { + found = 1; + break; + } + + p = next(p, arg); + } + + *pos = (p + item_size <= buf) ? 0 : (u32)(p - buf); + return found; +} + +/* + * Search a NT_GNU_PROPERTY_TYPE_0 for GNU_PROPERTY_X86_FEATURE_1_AND. + */ +static int find_feature_x86(struct file *file, unsigned long desc_size, + loff_t file_offset, u8 *buf, u32 *feature) +{ + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + u32 last_pr = 0; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < desc_size; done += buf_pos) { + read_size = desc_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) + return (ret < 0) ? ret : -EIO; + + ret = 0; + found = scan(buf, read_size, sizeof(struct property_x86), + test_property_x86, next_property, + &last_pr, &buf_pos); + + if ((!buf_pos) || found) + break; + + file_offset += buf_pos - read_size; + } + + if (found) { + struct property_x86 *pr = + (struct property_x86 *)(buf + buf_pos); + + if (pr->pr_datasz == 4) { + u32 *max = (u32 *)(buf + read_size); + u32 *data = (u32 *)((u8 *)pr + sizeof(*pr)); + + if (data + 1 <= max) { + *feature = *data; + } else { + file_offset += buf_pos - read_size; + file_offset += sizeof(*pr); + ret = kernel_read(file, feature, 4, + &file_offset); + } + } + } + + return ret; +} + +/* + * Search a PT_NOTE segment for NT_GNU_PROPERTY_TYPE_0. + */ +static int find_note_type_0(struct file *file, unsigned long note_size, + loff_t file_offset, u32 align, u32 *feature) +{ + u8 *buf; + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + + buf = kmalloc(BUF_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < note_size; done += buf_pos) { + read_size = note_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) { + ret = (ret < 0) ? ret : -EIO; + kfree(buf); + return ret; + } + + /* + * item_size = sizeof(struct elf_note) + elf_note.n_namesz. + * n_namesz is 4 for the note type we look for. + */ + ret = scan(buf, read_size, sizeof(struct elf_note) + 4, + test_note_type_0, next_note, + &align, &buf_pos); + + file_offset += buf_pos - read_size; + + if (ret && !found) { + struct elf_note *n = + (struct elf_note *)(buf + buf_pos); + u32 start = round_up(sizeof(*n) + n->n_namesz, align); + u32 total = round_up(start + n->n_descsz, align); + + ret = find_feature_x86(file, n->n_descsz, + file_offset + start, + buf, feature); + found++; + file_offset += total; + buf_pos += total; + } else if (!buf_pos || ret) { + ret = 0; + *feature = 0; + break; + } + } + + kfree(buf); + return ret; +} + +#ifdef CONFIG_COMPAT +static int check_notes_32(struct file *file, struct elf32_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 4)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +#ifdef CONFIG_X86_64 +static int check_notes_64(struct file *file, struct elf64_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 8)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +int arch_setup_features(void *ehdr_p, void *phdr_p, + struct file *file, bool interp) +{ + int err = 0; + u32 feature = 0; + + struct elf64_hdr *ehdr64 = ehdr_p; + + if (!cpu_x86_cet_enabled()) + return 0; + + if (ehdr64->e_ident[EI_CLASS] == ELFCLASS64) { + struct elf64_phdr *phdr64 = phdr_p; + + err = check_notes_64(file, phdr64, ehdr64->e_phnum, + &feature); + if (err < 0) + goto out; + } else { +#ifdef CONFIG_COMPAT + struct elf32_hdr *ehdr32 = ehdr_p; + + if (ehdr32->e_ident[EI_CLASS] == ELFCLASS32) { + struct elf32_phdr *phdr32 = phdr_p; + + err = check_notes_32(file, phdr32, ehdr32->e_phnum, + &feature); + if (err < 0) + goto out; + } +#endif + } + + memset(¤t->thread.cet, 0, sizeof(struct cet_status)); + + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) { + if (feature & GNU_PROPERTY_X86_FEATURE_1_SHSTK) { + err = cet_setup_shstk(); + if (err < 0) + goto out; + } + } + +out: + return err; +} diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index efae2fb0930a..b891aa292b46 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1081,6 +1081,21 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES + if (interpreter) { + retval = arch_setup_features(&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + } else { + retval = arch_setup_features(&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + + if (retval < 0) + goto out_free_dentry; +#endif + if (elf_interpreter) { unsigned long interp_map_addr = 0; diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index c5358e0ae7c5..5ef25a565e88 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -372,6 +372,7 @@ typedef struct elf64_shdr { #define NT_PRFPREG 2 #define NT_PRPSINFO 3 #define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5 #define NT_AUXV 6 /* * Note to userspace developers: size of NT_SIGINFO note may increase From patchwork Thu Oct 11 15:15:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B411112B for ; Thu, 11 Oct 2018 15:22:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09FF92BA40 for ; Thu, 11 Oct 2018 15:22:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F1B4C2BA3B; Thu, 11 Oct 2018 15:22:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 703892BA36 for ; Thu, 11 Oct 2018 15:22:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9D906B0282; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9A9A16B028C; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B30B6B0290; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id BA1026B028F for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id i81-v6so8148113pfj.1 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=w0kT9LujSLvc82kqhfwMcvEMstJT4PvNNrcS249mWtg=; b=Da1udrjXXpOZU1pz8D/Ckh06cz83q/TmJzEfQ2gzoLueyf3XCwsCkkBx4GkQf+eoil MMqKEEfzwY6M8zZgjxEIbNuLAAhUGid3AyTxihits/6aMVVkUNxc6WLzo52JOgGIFYG9 JuGMWjt3krNuxhkKNJUt7y835IoFKgbhES4mC3OEU8AAqh/vyadsyQoCyHRsm2PEmIzJ nvrd8dFzwxd5IB067aXFzUmukTCJfM9h+D2MKs5bfB4f12mQ5w5PpbYeegqHk1Oc+MRD pGmIWN7MlrumkRAM50Erw4j3dFoMARMJyo7TpYDklBrMZUpIsy62nJDydSwdTintTWHC AkQA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfogXpoNhCxkUTJqofARDj5kypw9BFFMKCZ1NXpuxBblWl4gumQ+b zcstNfdz4FfDUAJRC7cKqmf6q7pS1q3KZKFTGis8bl8QDyhUJzGVE/wU6qT0UWezFnUKFv2AZmS k50FxIInrZzTg3FXijYc6VbPJ5U44kgdbTPil+8Q6XGdHb4KXpHSGyWuBah2fUfhiOg== X-Received: by 2002:a63:f448:: with SMTP id p8-v6mr1827897pgk.124.1539271261402; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-Smtp-Source: ACcGV61LonuUCfKu8sv9Ftg5dfInu/h26X8Stz1x4223AXhs0TOHnqo1yJCH/dxFQw+OWaMuHHlB X-Received: by 2002:a63:f448:: with SMTP id p8-v6mr1827853pgk.124.1539271260540; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=blfYrN28h+lMTXi18DTlofBkrHuHAnE/OJhn54nNXoEjmXFSTxVSluIJOTKik4Yq3v fJOpiVr2yihcaVot926GlzwLitkwIsLn9YqlBhxYYQeQwwxDPgZfyLLBmWxDKKVNuPR0 U3fSvUamEzFFyehWI93gBIM+IdYEw1eKf5tBgz+TZ7KdPSLbTD+mPM3ybCmsmUpFBNos iqnbXowo3w9WBcVAaZl/1yCiFUB6y/TCYzP9kqu6yygcWvmJpZq8oRHz0On8A5GX5mQF cADyRQlEEw09HelEmZUEcaQ9fs7grGz4ywl/k2pRsfykbBIPRd4IjrKc7vN/m6PE/QH9 SQkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=w0kT9LujSLvc82kqhfwMcvEMstJT4PvNNrcS249mWtg=; b=z9GbW/78BJKEwEOR0t6aTa3IiN8V92rqqiiBlYEtPYpNmXeWl3P+uJmCUhTNThUx0s 3wm0muLUTzibH6wDIH9uTIzmH3e2DPbmjYUNPILvLaWtJc1cHAm3mfPWSx9YFQx6JjDP dGjnEI1yQ0vxDQo/ef2t/4/hrHUyv1vykxVOJS+dbx0NMXSPCSBJodr0vNHU8QHLUJr5 0+rATSjvORfcbZEor+aQpPF/IW0Zw1voKZsD0FGhCpewx5xFxzKYDAa+sd6d3dtVHsoa TFst4oBj5khbncb6YLCEPjkIf565+yenIy2tqXIz4TBGtIGYyiu7R+PY2+beG2J/kfnJ tnqw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id bg10-v6si26471290plb.47.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019169" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 24/27] x86/cet/shstk: Handle thread shadow stack Date: Thu, 11 Oct 2018 08:15:20 -0700 Message-Id: <20181011151523.27101-25-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The shadow stack for clone/fork is handled as the following: (1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child. It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK. (2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK. (3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret. This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 34 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 ++++++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 3af544aed800..008b4651b5be 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -17,12 +17,14 @@ struct cet_status { #ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk(void); +int cet_setup_thread_shstk(struct task_struct *p); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline int cet_setup_thread_shstk(struct task_struct *p) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index eeeb9289c764..8da7c999b7ee 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -223,6 +224,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 2000e5e87f8f..51b26616ac20 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -145,6 +145,40 @@ int cet_setup_shstk(void) return 0; } +int cet_setup_thread_shstk(struct task_struct *tsk) +{ + unsigned long addr, size; + struct cet_user_state *state; + + if (!current->thread.cet.shstk_enabled) + return 0; + + state = get_xsave_addr(&tsk->thread.fpu.state.xsave, + XFEATURE_MASK_SHSTK_USER); + + if (!state) + return -EINVAL; + + size = tsk->thread.cet.shstk_size; + if (size == 0) + size = rlimit(RLIMIT_STACK); + + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + if (addr >= TASK_SIZE_MAX) { + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + tsk->thread.cet.shstk_enabled = 0; + return -ENOMEM; + } + + state->user_ssp = (u64)(addr + size - sizeof(u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; + return 0; +} + void cet_disable_shstk(void) { u64 r; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 4a776da4c28c..440f012ef925 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -125,6 +125,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + cet_disable_free_shstk(tsk); fpu__drop(fpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index ea5ea850348d..9cdbd87bb908 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -325,6 +325,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp, if (sp) childregs->sp = sp; + /* Allocate a new shadow stack for pthread */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM) { + err = cet_setup_thread_shstk(p); + if (err) + goto out; + } + err = -ENOMEM; if (unlikely(test_tsk_thread_flag(me, TIF_IO_BITMAP))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr, From patchwork Thu Oct 11 15:15:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636891 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EBB7617E1 for ; Thu, 11 Oct 2018 15:22:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D82CE2BA37 for ; Thu, 11 Oct 2018 15:22:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC0A62BA33; Thu, 11 Oct 2018 15:22:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C3722BA21 for ; Thu, 11 Oct 2018 15:22:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3E246B028F; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6473D6B0285; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BE766B0285; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 6CE7E6B0276 for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id s15-v6so6275016pgv.9 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=V56ObHxpVBzvq9rlMW5IthWJ0mE9AdzBQaFBjAUSUvE=; b=rdVz+Y3mud67sAQ5zA7Ff2yk2vBXjWXfZHql3YWJc/sfxf5JBzUf68F43s+vk1DOkj EHIgh028bS/2xsN+zK1qWMP1D9WD8Yo3/tATnON7H0ph+XJaJhZswzwbKKdNf94pjMCe TKzfRVM1Wa4K6gb3vJJKbohrr+fMqB5zjkvw4WZjSWGb1VIRK3oFULLW0lMbUhWZ2+Jr wfg26V6cVjumbgXxH6uXCtj0tDQ8pue38NWxHwJj3DA0qJizG5Iw/ktQN8Z1+6pGfOox jsUlma/W7BBaKhcmWBbSk6oMoCyQtMzEiGFxjk7/XIlRysVg63Mb3yyMQy6TvSLbGuQL fyBA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojzsDgaqc1Atg4jvPb6hlTDOJ5yB5nHw6Q9HD+A3p5GIl/3m1DT pTXJIfyHUqfNmBWh958aHCKCU1k3g9AEQGjnSSylHw1uPEtSN+Dqw82+6Bch1PSXsZ7rRvSakN/ 7E0dQXlPcYPl6flPYWI7L72hL6DdPpKdJYs7PX9cW28m6MnY8PhW6u7R9IAHfxiFb1g== X-Received: by 2002:a17:902:6b03:: with SMTP id o3-v6mr1999834plk.333.1539271261137; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-Smtp-Source: ACcGV60aq797aLrvWyfwlwaREoVqvS/zGAQeziJA7u5qgi44WDv/QxqLin8J3Se275GKbQDoa9cp X-Received: by 2002:a17:902:6b03:: with SMTP id o3-v6mr1999809plk.333.1539271260529; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=SqgaMWxokRIaAb6PpKMfR6BjPN1fPjd4k1xQ4ZTI/Rn4DXmG/ecHnX8mmp9VFqz/YE wmx/AsGT2WVNAXOxZNPgu1x3ApRieM5hVTlRf1Q8ZcA+OYQEzZson767yLRgckP9CRPY FVre7K5+r1qjua7IKKuESEnJtcRRxPP1506Yr/cnj7QRpriuWeIhEAevbMmMCS/Z7Tol +cSHLHG3CzeAyPL7Q4mVK/5LgpliqEkuLJOWmIXjEuPkPMCllNzf7jR2yDMNz3oNAEcR tAPkc7k9MtdaHSXGjduTiFQm55hk8jVvysNoF3f0rS0N3O2ebUSl1AMh9+Nir5Dyg3UM OWpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=V56ObHxpVBzvq9rlMW5IthWJ0mE9AdzBQaFBjAUSUvE=; b=ZKrr+9MDY+2/a70Ypa48AeQ9KjOGpxlQ0GOouLkZuZdr0dAx3SGCqb6BDYtsNy5nZu 8PVu65xam1alKxSTRv/LJOSzn2M+aEhfharWiclABznH6P5MQkQA7+NIFQffBhtwq22b 30eiVkn0cTY0rVuFdaVZFmpmBltUd81ETAdIPplH9gMI/TH6igNLNJHJicpjmcwEDMcT I9ots5aTGvp/qxzSZYF2JfDBGg5RtqQ66Kt/JI3hx6H40MPfo+qVlR5CRlONV3t6G0as 5RQM9ag64ONJBoXiLTm2W/8Hv6vpmtzUBpdItR/cNOoihBjSroH3ME7IH8ASBcmZp0QW 9C7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id y62-v6si29748088pfy.139.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019172" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 25/27] mm/mmap: Add Shadow stack pages to memory accounting Date: Thu, 11 Oct 2018 08:15:21 -0700 Message-Id: <20181011151523.27101-26-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add shadow stack pages to memory accounting. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b85292014ec9..fa581ced3f56 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1677,6 +1677,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) if (file && is_file_hugepages(file)) return 0; + if (arch_copy_pte_mapping(vm_flags)) + return 1; + return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE; } @@ -3267,6 +3270,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; + else if (arch_copy_pte_mapping(flags)) + mm->data_vm += npages; } static vm_fault_t special_mapping_fault(struct vm_fault *vmf); From patchwork Thu Oct 11 15:15:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 57CC4112B for ; Thu, 11 Oct 2018 15:22:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43FF52BA44 for ; Thu, 11 Oct 2018 15:22:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 37D052BA26; Thu, 11 Oct 2018 15:22:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E44D2BA43 for ; Thu, 11 Oct 2018 15:22:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3362D6B0292; Thu, 11 Oct 2018 11:21:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CBB9C6B0291; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93A146B0282; Thu, 11 Oct 2018 11:21:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id E8D306B0291 for ; Thu, 11 Oct 2018 11:21:01 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id ba5-v6so2597102plb.17 for ; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=snzAwRlh6vkWcwqpDUm+EIVw1ndZvYFPzK/04CSGmDk=; b=axhu373R2HS5KHek1ymTYPDojqOhFE/aK3ljClrfdCv8rtKVT/KbMCUsH3CxzO1U0s zYKMijOECnygZDeJR6p/szUeGNAbzjZMcLguDc1e5HcdWaGDv8HkDLLsuBPiPNALuSDs wwR7zeJPAYxmuA/uGV8LggBN4gbNPaJqFgSvMnNfCvvEh61E0q43QtP7BFqtgRLCQkW6 VWCiOcP5L+P7uRAAed8bwzP/cA87nzeM6eR7pgENcD+I1ivxEumotC5UK8LSfqS+ww5q bwY1ddPbhBbQvUpYJf+5eXCOt4wQJ7g7W21KNjmnnPkDdmdHB5g3OOb4eVJNvnIf+jrN WD3w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoheb1v5+oqAlLIzpjgsp1tE12CFqBZBkvBJWlS5lc/FDrx5vtIk DwJ8h5gAhroRTLwC9aLAh8zt05I34DQAHacZxRWpo/GhTOX6IO3r7MXMSl6wCLYnoJ2Fx2W0NZt kXiYuWf2Ovm+NkZ93hpHYdvYzUuVc5iywfWkc1LWJaWKZA3f3ftrtzSNcY+zs8UoS7A== X-Received: by 2002:a17:902:20c5:: with SMTP id v5-v6mr2010969plg.62.1539271261607; Thu, 11 Oct 2018 08:21:01 -0700 (PDT) X-Google-Smtp-Source: ACcGV62a2jlZcxHeO0fVNK7Yw2w1CjlwC/E/sTE5QUP2cJhXmzlVWPYJ7MWRJd/1eJ7RMDHAovXI X-Received: by 2002:a17:902:20c5:: with SMTP id v5-v6mr2010912plg.62.1539271260691; Thu, 11 Oct 2018 08:21:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271260; cv=none; d=google.com; s=arc-20160816; b=w64R1Ykmoo1HSlLgnk4Wa+lXF3MBd1J98YFjOVv5xIOphfcrc+ySNyzynDJF6CU6eD GdIbC/OvKYPUDNbe7yiQC/XAe/kWFkyYOLh01V99o/2jfjx46JRrqX9eM6w+8ebdEV0Y QmRasOrYDGLwC8KX33czufaUG0c7gVrOBHLMJ4W+JGSYmpxE2FTX4nA8r2wO2TgWBo6e dV4mIPEDx+EdnV4OGrnL7NrsEAiuQzoo/W4RWAl3XJVnAUV3s9bmMqOSnh8AXYMBwDuW HVQgk0KWGHBLyfUwgr/kxb5v6g1FpTY0Xi6MrY9JpWkbkpkVOYBNY5XiH6fd2Rc2cypw 920w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=snzAwRlh6vkWcwqpDUm+EIVw1ndZvYFPzK/04CSGmDk=; b=tWxmMDKmPs3ENlyjKXIaat807Jdi6ARGEb2hqxFWIWkEqp0T0OuHSKuZ4OotyCHTVQ 4hNDsIg9pg8Um91pIgRMsao8ORShO/8ho+aHThVT19hNuG8HZwedagOh3TS0+t2or1qP LItyBRWrAwaT19+0fPNKJVqQ/QXBHjmdm0akdMiCZ6rJRdsrfZS3Vm3KSZIg89IayBtL LLBw5XmzP3yjq7NaUO6Lh+vgr0sF9qAV1SPo4OELrsqTKZYk5ldehwlq8Q3tVfXiIHhm 0dIVn/Uu3mwwR1GdVWl/Bwmdm+9qq+aKlDLx0FlzZvmqLmWHiZZFx0dFESZKoVxce+Vu Iimw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id g9-v6si6571810plo.328.2018.10.11.08.21.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:21:00 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019181" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 26/27] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Date: Thu, 11 Oct 2018 08:15:22 -0700 Message-Id: <20181011151523.27101-27-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) Return CET feature status. The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information: *addr = SHSTK/IBT status *(addr + 1) = SHSTK base address *(addr + 2) = SHSTK size arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) Disable CET features specified in 'features'. Return -EPERM if CET is locked. arch_prctl(ARCH_X86_CET_LOCK) Lock in CET feature. arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) Allocate a new SHSTK. The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK. There is no CET enabling arch_prctl function. By design, CET is enabled automatically if the binary and the system can support it. The parameters passed are always unsigned 64-bit. When an ia32 application passing pointers, it should only use the lower 32 bits. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 29 +++++++++++ arch/x86/kernel/cet_prctl.c | 86 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 4 +- 6 files changed, 128 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cet_prctl.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 008b4651b5be..6fa23a41580c 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -12,19 +12,24 @@ struct task_struct; struct cet_status { unsigned long shstk_base; unsigned long shstk_size; + unsigned int locked:1; unsigned int shstk_enabled:1; }; #ifdef CONFIG_X86_INTEL_CET +int prctl_cet(int option, unsigned long arg2); int cet_setup_shstk(void); int cet_setup_thread_shstk(struct task_struct *p); +int cet_alloc_shstk(unsigned long *arg); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else +static inline int prctl_cet(int option, unsigned long arg2) { return -EINVAL; } static inline int cet_setup_shstk(void) { return -EINVAL; } static inline int cet_setup_thread_shstk(struct task_struct *p) { return -EINVAL; } +static inline int cet_alloc_shstk(unsigned long *arg) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 5a6aac9fa41f..d962f0ec9ccf 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -14,4 +14,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_X86_CET_STATUS 0x3001 +#define ARCH_X86_CET_DISABLE 0x3002 +#define ARCH_X86_CET_LOCK 0x3003 +#define ARCH_X86_CET_ALLOC_SHSTK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 36b14ef410c8..b9e6cdc6b4f7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,7 +139,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 51b26616ac20..17ad328586aa 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -121,6 +121,35 @@ static int create_rstor_token(bool ia32, unsigned long ssp, return 0; } +int cet_alloc_shstk(unsigned long *arg) +{ + unsigned long len = *arg; + unsigned long addr; + unsigned long token; + unsigned long ssp; + + addr = do_mmap_locked(0, len, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + if (addr >= TASK_SIZE_MAX) + return -ENOMEM; + + /* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + + if (!in_ia32_syscall()) + token |= TOKEN_MODE_64; + ssp -= 8; + + if (write_user_shstk_64(ssp, token)) { + vm_munmap(addr, len); + return -EINVAL; + } + + *arg = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c new file mode 100644 index 000000000000..320dbb620d61 --- /dev/null +++ b/arch/x86/kernel/cet_prctl.c @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* See Documentation/x86/intel_cet.txt. */ + +static int handle_get_status(unsigned long arg2) +{ + unsigned int features = 0; + unsigned long shstk_base, shstk_size; + unsigned long buf[3]; + + if (current->thread.cet.shstk_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + + shstk_base = current->thread.cet.shstk_base; + shstk_size = current->thread.cet.shstk_size; + + buf[0] = (unsigned long)features; + buf[1] = shstk_base; + buf[2] = shstk_size; + return copy_to_user((unsigned long __user *)arg2, buf, + sizeof(buf)); +} + +static int handle_alloc_shstk(unsigned long arg2) +{ + int err = 0; + unsigned long arg; + unsigned long addr = 0; + unsigned long size = 0; + + if (get_user(arg, (unsigned long __user *)arg2)) + return -EFAULT; + + size = arg; + err = cet_alloc_shstk(&arg); + if (err) + return err; + + addr = arg; + if (put_user(addr, (unsigned long __user *)arg2)) { + vm_munmap(addr, size); + return -EFAULT; + } + + return 0; +} + +int prctl_cet(int option, unsigned long arg2) +{ + if (!cpu_x86_cet_enabled()) + return -EINVAL; + + switch (option) { + case ARCH_X86_CET_STATUS: + return handle_get_status(arg2); + + case ARCH_X86_CET_DISABLE: + if (current->thread.cet.locked) + return -EPERM; + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + cet_disable_free_shstk(current); + + return 0; + + case ARCH_X86_CET_LOCK: + current->thread.cet.locked = 1; + return 0; + + case ARCH_X86_CET_ALLOC_SHSTK: + return handle_alloc_shstk(arg2); + + default: + return -EINVAL; + } +} diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 440f012ef925..f240fce2b20f 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -792,7 +792,7 @@ long do_arch_prctl_common(struct task_struct *task, int option, return get_cpuid_mode(); case ARCH_SET_CPUID: return set_cpuid_mode(task, cpuid_enabled); + default: + return prctl_cet(option, cpuid_enabled); } - - return -EINVAL; } From patchwork Thu Oct 11 15:15:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10636869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74D39679F for ; Thu, 11 Oct 2018 15:21:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60DFC2B9E9 for ; Thu, 11 Oct 2018 15:21:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52C512BA19; Thu, 11 Oct 2018 15:21:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 929492B9A6 for ; Thu, 11 Oct 2018 15:21:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F60E6B026F; Thu, 11 Oct 2018 11:20:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 476576B0270; Thu, 11 Oct 2018 11:20:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31BFB6B0271; Thu, 11 Oct 2018 11:20:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id D5E7C6B026F for ; Thu, 11 Oct 2018 11:20:53 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id f17-v6so6536885plr.1 for ; Thu, 11 Oct 2018 08:20:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=TGy/yUeZqb9rLV1mubnW1ez/uQK99/Tli2aQrk16AmN0GbCPlLIgKovE3XQq7TUi42 fogACEpul2R/xO0jj2FngQLVRnGSJ8GbvWVqXpIa+mYZ7TBFIGm9lw/m0094BZ9rOX/i nVaknZhXb8qxLzRRnrGZ6dn1pAjGZjxjUQxszQaxgZrOaCrU0gb9Zps2MKg219xXlFPc QbkFycDaC4msoSNWbJwt8OMS0qAhR3Gi78dvz/pHEbj4UF8Oi77w+aIcjHWUwFiPsmf2 bwiyayuxOpwAot5eljsx5pL8gMRGLN7/Q969KC+BIsjorjichcZ/A/tSiW/HGeSqIY7P 5ysw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojz6FR+51yRrC2oqNZja17t9N6HRzdcMlKERCtsLsx6eE91tQZw ga3iUJf31rA0ywE/6xTBZXW6I3+EdXYILNTyyN1Wl2hBnFTX68U59fQN5JavwyTr4iDk4GOdp5Y tmLN4JdpTy5e5RyENIYpdvTbwR/5vf7kuQ06c+jUKxfgGX7Zgne9O2St/meFwiy4tmg== X-Received: by 2002:a62:2315:: with SMTP id j21-v6mr2048229pfj.90.1539271253526; Thu, 11 Oct 2018 08:20:53 -0700 (PDT) X-Google-Smtp-Source: ACcGV62zMPW7ytyWZ0NE+3gFXBbJVAf4OyuGLbP7S3q9Ps1it7GI0UHElKH4QW04Jnh7eQIVlccp X-Received: by 2002:a62:2315:: with SMTP id j21-v6mr2048147pfj.90.1539271252493; Thu, 11 Oct 2018 08:20:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539271252; cv=none; d=google.com; s=arc-20160816; b=oRT1vK0keOHV5Zpwu4lA7LAxkY4IjR8MFnTgJdCpbf/nCpidvuz0G9Y1EETmv52D3H OK8n1gF0jtO4BlJ44Sbrcr0+nHPKNyXCvMMqjXq4HcTWlN6dJzhQRJrdFCDGUqq9afLz oyOte4EOHTiFv67HM0FgmTdJNgXnHBh+zELAv4ZipWWZ3jbCGJUjazai83EgBp76mFE+ 23LIhU2XuC7cXherbVUvSVZamOSs0KPnEprLALPDLjFKxcw9KeEcai11n95nxNISNnlT ZjvHyPgv3x9MoQMJ9oDnKNX8cjyiF5G3P5PNxW6GBSDr/eBxYpk7RMRHsCBGriJU6zyL 9AXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=GRheRjIGz99clKHYAMu+D8sAOLtLwoDLPGh8EQzamcObfylG+L5bii8D9YJ78vliL7 BtPTm8c7+fvXDCWvi12nKOVlCBVZNIOuwVwGSrhoyG+CcRQHXUE8Sha28NkJ1cDNAS14 vZFSENfBNv9Okwl/vQ7IxoTdDabHRgSbET9GvA1tkYdWKSknRIkuaWZJ4dBZbOTg1RnF dvDPxq87mRGXYknOLXpOevuOv7f5Z7nhH1xiKMgmmL5WWG+6HOlUexbtFQTHzK0xClsa gYtBI3Oo2m/VPbdUrnGhO2J7YDDsA2kYemaWCHsXHfOEBVBLd9DJtL9lFOX6UIAV+icK 2CjA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id i9-v6si27987528plt.111.2018.10.11.08.20.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 08:20:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 08:20:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,368,1534834800"; d="scan'208";a="78019185" Received: from 2b52.sc.intel.com ([143.183.136.147]) by fmsmga008.fm.intel.com with ESMTP; 11 Oct 2018 08:20:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [PATCH v5 27/27] x86/cet/shstk: Add Shadow Stack instructions to opcode map Date: Thu, 11 Oct 2018 08:15:23 -0700 Message-Id: <20181011151523.27101-28-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181011151523.27101-1-yu-cheng.yu@intel.com> References: <20181011151523.27101-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the following shadow stack management instructions. INCSSP: Increment shadow stack pointer by the steps specified. RDSSP: Read SSP register into a GPR. SAVEPREVSSP: Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack. RSTORSSP: Restore from a "restore token" pointed by a GPR to SSP. WRSS: Write to kernel-mode shadow stack (kernel-mode instruction). WRUSS: Write to user-mode shadow stack (kernel-mode instruction). SETSSBSY: Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR. CLRSSBSY: Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable