From patchwork Thu Aug 30 14:38:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581829 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8E4B5A4 for ; Thu, 30 Aug 2018 14:43:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D79C12B560 for ; Thu, 30 Aug 2018 14:43:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CAFCE2BC25; Thu, 30 Aug 2018 14:43:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 542642B560 for ; Thu, 30 Aug 2018 14:43:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D54AB6B521C; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2C936B521D; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA0526B521E; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 688C96B521B for ; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r130-v6so2461809pgr.13 for ; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GItZ6zMjewsshyErngy5TzAWyns2xf+Aa7z7tr61mP8=; b=MlufohkZU3NNtS6vRzQxmApPtijzAD6qkQdyvwPLP6GRqxCT83WjCo2RQiaxhtaCIV UVmS2juMmlu6M6vTHpKsT0MD9heJURImKqNbbi95JYLqc3Vwz4HxvNMJW8RZpgB82Nfi Ct4vPOy9YHPnqADSpjkiM26cXuomlKRweV+uL1SqSW9UQmhBHDbabfW/N+kT5F006Jmw 3gje2Ywbz58IBmxdi3Av5+gyw/0VHVxHGwDflALLuo2l8W8PkbyA+0LRqjaOCJeumOov 7QrEcBS2z/wvo1xM2OgOJMU5bkXj8Jat0CojR83JYN5RZwQ1rjNcsRC4p6h34fLUr++x YEQQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DUawyf+iNZWnVQ/umOt4s1SBgw0CzXjlug/nl6JObMq7nm8hNw dfXsHFe2VwRIW0vFv9OD+mDpwKw053g67GU8t+pamHw5gUvea7GQk0LUtOxb7qfmhhJqOthEGoa hp84v0ceUfwlnvByqDFFkj3ndDJEIXWGWcXinA9TIKkSP8tTWFXNRqzH5pLWjwdJsAw== X-Received: by 2002:a63:e914:: with SMTP id i20-v6mr10072206pgh.10.1535640225100; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb6NGXeqEVRGEYKspXinzamcjMYwh2yg1S98UylXuuWxCUM1kKFDb8/Ka2Qp99y7hm8OVwl X-Received: by 2002:a63:e914:: with SMTP id i20-v6mr10072175pgh.10.1535640224453; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=NGv1Yfx3Hf09ohpfJkx4bCikmutELXbapBwDwXVKXGDkSZ6uBs5Myhj4sj3qVpyKBa x7VDlWM0Ga4krxID5p1wv6QWTwJIBoUQ5fkxTW7OW+dC3Pw8x0IqEXlzT0sGeNbmPMLB amtuGpklVPZ/7oAhfTbQnSUy7LGpv4ceAH9O9vaDUBJR9hP2c0It8uJueDkv8s6KgHuL QuVk81oVHuUBLO3x0z3BTvjgZ/IXdPKk6W+d4cgJXLS1aqr6r49JOfMViLhScjbZc7Of UqeuBkXvMr0494aqpZdhiT8k1At97SRWFIoxu/9Yf1OoC0P5ejLp2Z/KJqhfXfHrJX86 pQTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=GItZ6zMjewsshyErngy5TzAWyns2xf+Aa7z7tr61mP8=; b=Vu5GNiUv5g/SLse0c4FlLGMNI+O1R5yIxPlZaFGt+c+nEZDvyeFnbTbSTEHJYDGLhK 3O+DDv3IpkiN/qyqlAKGt07KA4yzBNTunwjiK0OMqNimB8rvY1ZhXH0Ji4UWw+M9W9q7 KGrADWxkfNOzJaP9Y67jIexQel5qBicXpV2aMsAFHg42034WFUxRsIvFxqbSE7Z5V9Fc 7zgCrkqZc6urX+SHVpS63NLh6HUDzWDl1SDoZc+AtofK9akNuf8wVPLtlHvYuBIiCilJ zHV/vstilGTP+mCXKtoTS0HBIr8QTM210FC9Z2T8ts1arvqX+qezhtKgJNBpdPCoUJtj 4yAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186662" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:39 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 01/24] x86/cpufeatures: Add CPUIDs for Control-flow Enforcement Technology (CET) Date: Thu, 30 Aug 2018 07:38:41 -0700 Message-Id: <20180830143904.3168-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add CPUIDs for Control-flow Enforcement Technology (CET). CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 3 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 89a048c2faec..fa69651a017e 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -221,6 +221,7 @@ #define X86_FEATURE_ZEN ( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */ #define X86_FEATURE_L1TF_PTEINV ( 7*32+29) /* "" L1TF workaround PTE inversion */ #define X86_FEATURE_IBRS_ENHANCED ( 7*32+30) /* Enhanced IBRS */ +#define X86_FEATURE_IBT ( 7*32+31) /* Indirect Branch Tracking */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ @@ -321,6 +322,7 @@ #define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c index 772c219b6889..63cbb4d9938e 100644 --- a/arch/x86/kernel/cpu/scattered.c +++ b/arch/x86/kernel/cpu/scattered.c @@ -21,6 +21,7 @@ struct cpuid_bit { static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_APERFMPERF, CPUID_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CPUID_ECX, 3, 0x00000006, 0 }, + { X86_FEATURE_IBT, CPUID_EDX, 20, 0x00000007, 0}, { X86_FEATURE_CAT_L3, CPUID_EBX, 1, 0x00000010, 0 }, { X86_FEATURE_CAT_L2, CPUID_EBX, 2, 0x00000010, 0 }, { X86_FEATURE_CDP_L3, CPUID_ECX, 2, 0x00000010, 1 }, From patchwork Thu Aug 30 14:38:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581849 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8744C14E1 for ; Thu, 30 Aug 2018 14:44:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74F1F2B560 for ; Thu, 30 Aug 2018 14:44:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67B892BC83; Thu, 30 Aug 2018 14:44:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19F6E2B560 for ; Thu, 30 Aug 2018 14:44:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C57AD6B521E; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6C0546B5221; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDA346B5221; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id BF06B6B521E for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id b93-v6so4055932plb.10 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=kYlQ1ATYPX18BODPZ6MDhmQ3OmDqE0vEbn/QaIqn05k=; b=S0QG9vcssJBYimPEEhsxyRPUozXdZDyBLnnYC4rpHDNyyWlB6vfTvgT5pJTn3kLHPb OgeKt9P/N8RB8JwHT5Qeq+Q8Z5wWw7Skc6yFnNClMiDrJ5iGAWivsNJKCIii3yuC7o3/ Lowc308/kzU/ECSOLWl25xav1JESVtDfe5GiXHZ2DeaOoiKaG7GjLRbnX3y/TW4HkosU 2OfqG7qtWg6TQcnZryliVV75ppBS61yCSbhB6HRq3EnujaWRobyBsATtdgsyaCS+/wTs /YpssGiRXOtctu5Qn/+2ZLuYfRUmvVvTS8yGw/fsr5R4F0FheQV2iD5DXJyyiCC0O7Uo ikzg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51B3plYKoLVsTlnY05idltyhchXCynFLypvmRzSLNzWKHZX3OXu4 i7G2i7G4ZUxxIP18vm9rrTHHkMmY8tIsPQTOVoVX8y0WuVNsU8eJwOmwVDwotvQRzmwCAIXz+Ud 92DoQHYzdu5u5sBmTQ89jj293BVTpm/+zHIVIZwmjej0kc9DSPAa9geLEGSfG/dusFQ== X-Received: by 2002:a17:902:b58a:: with SMTP id a10-v6mr10768394pls.306.1535640226432; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda2Hxq4E73H0Mq4aqLKOMxlaQ4JUD33bF72LA/yBZvMwknlKhbzgmACfJq2PHjMUlViG3Y7 X-Received: by 2002:a17:902:b58a:: with SMTP id a10-v6mr10768326pls.306.1535640224976; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=fBHqViwqWLzg5EInc66OnI82skBxFjJ1svy5I4eKL+CBYt43yVnftt0Aw/QxoN5tIj gx9hfFdPgaiEo4LlNhH4U5Ck9Pc99TMg31EyuE3o2vwwkSqfVfTRyIdor98KZM4llJRs 4/AlVL7hExznRrihGrg76ci3XaunmkbfCphXR/K5Qdhk3xonrkm+P2DqlSop0YnAgTTt ywEnjGqkC3WTZ6EG2sVEqcql1Y3py0zf/6nafpmhd9vZB3XQkq7KEvvY2Uckt8EVRKGV MMhdhX2V0O2TSyJNakJ+kwVP9VdONr1qKP6Z2ifUz484oR/5rUfduR3953noRIXI5nMM QNPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=kYlQ1ATYPX18BODPZ6MDhmQ3OmDqE0vEbn/QaIqn05k=; b=0Q2LAlZ+hAsEGfQ622vdA4fRsTLGKMkzV8S8ugAy2UZ2Rd/4hNGX5Ai0H7MvqmwTJc PA8ULtGserejLNrPt6SVatyrhI3GSP8BdjgtiuaCoMy3iQfkPXGCJnSd28mbxE5tXMzL lI2fuJtg28itdIK/jP/C3D8qnzdk8IENgQhkRua+CDeRiM2AW8HszwFvrgYuIYdGiE53 8rw2gYC0VvtT+6MJuxpiB23jI71vBjVCtwWHCFV9YYrslDhGYMlmzqOdrQujL8qRpBas 1kO9+17PTgbrdFTf1fS7Ku9nJuEKRSXo7ZrSTChBgCSNCRFA5j9x7DnG/rEDrqUn3Tbo gecQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id t2-v6si6809115pge.64.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186665" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:39 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 02/24] x86/fpu/xstate: Change some names to separate XSAVES system and user states Date: Thu, 30 Aug 2018 07:38:42 -0700 Message-Id: <20180830143904.3168-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To support XSAVES system states, change some names to distinguish user and system states. Change: supervisor to system copy_init_fpstate_to_fpregs() to copy_init_user_fpstate_to_fpregs() xfeatures_mask to xfeatures_mask_user XCNTXT_MASK to SUPPORTED_XFEATURES_MASK (states supported) Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 24 ++++---- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +- arch/x86/kernel/fpu/xstate.c | 88 +++++++++++++++-------------- 6 files changed, 66 insertions(+), 63 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index a38bf5a1e37a..f1f9bf91a0ab 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -93,7 +93,8 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | + xfeatures_mask_user; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -233,7 +234,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) /* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact - * format and supervisor states in addition to modified optimization in + * format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 48581988d78c..9b382e5157ed 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,19 +19,19 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) +/* System features */ +#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT) /* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " @@ -39,7 +39,7 @@ #define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 2ea85b32421a..4bd56079048f 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -363,7 +363,7 @@ void fpu__drop(struct fpu *fpu) * Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs(void) +static inline void copy_init_user_fpstate_to_fpregs(void) { if (use_xsave()) copy_kernel_to_xregs(&init_fpstate.xsave, -1); @@ -395,7 +395,7 @@ void fpu__clear(struct fpu *fpu) preempt_disable(); fpu__initialize(fpu); user_fpu_begin(); - copy_init_fpstate_to_fpregs(); + copy_init_user_fpstate_to_fpregs(); preempt_enable(); } } diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 6abd83572b01..761c3a5a9e07 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -229,7 +229,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ u64 __init fpu__get_supported_xfeatures_mask(void) { - return XCNTXT_MASK; + return SUPPORTED_XFEATURES_MASK; } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 23f1691670b6..f77aa76ba675 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -249,11 +249,11 @@ static inline int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_ { if (use_xsave()) { if ((unsigned long)buf % 64 || fx_only) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_fxregs(buf); } else { - u64 init_bv = xfeatures_mask & ~xbv; + u64 init_bv = xfeatures_mask_user & ~xbv; if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_xregs(buf, xbv); @@ -417,7 +417,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 87a57b7642d3..19f8df54c72a 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -53,11 +53,11 @@ static short xsave_cpuid_features[] __initdata = { /* * Mask of xstate features supported by the CPU and the kernel: */ -u64 xfeatures_mask __read_mostly; +u64 xfeatures_mask_user __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +82,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -113,14 +113,14 @@ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) } EXPORT_SYMBOL_GPL(cpu_has_xfeatures); -static int xfeature_is_supervisor(int xfeature_nr) +static int xfeature_is_system(int xfeature_nr) { /* - * We currently do not support supervisor states, but if + * We currently do not support system states, but if * we did, we could find out like this. * * SDM says: If state component 'i' is a user state component, - * ECX[0] return 0; if state component i is a supervisor + * ECX[0] return 0; if state component i is a system * state component, ECX[0] returns 1. */ u32 eax, ebx, ecx, edx; @@ -131,7 +131,7 @@ static int xfeature_is_supervisor(int xfeature_nr) static int xfeature_is_user(int xfeature_nr) { - return !xfeature_is_supervisor(xfeature_nr); + return !xfeature_is_system(xfeature_nr); } /* @@ -164,7 +164,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask) == xfeatures_mask) + if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) return; /* @@ -191,7 +191,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * in a special way already: */ feature_bit = 0x2; - xfeatures = (xfeatures_mask & ~xfeatures) >> 2; + xfeatures = (xfeatures_mask_user & ~xfeatures) >> 2; /* * Update all the remaining memory layouts according to their @@ -219,20 +219,20 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) return; /* - * Make it clear that XSAVES supervisor states are not yet + * Make it clear that XSAVES system states are not yet * implemented should anyone expect it to work by changing * bits in XFEATURE_MASK_* macros and XCR0. */ - WARN_ONCE((xfeatures_mask & XFEATURE_MASK_SUPERVISOR), - "x86/fpu: XSAVES supervisor states are not yet implemented.\n"); + WARN_ONCE((xfeatures_mask_user & XFEATURE_MASK_SYSTEM), + "x86/fpu: XSAVES system states are not yet implemented.\n"); - xfeatures_mask &= ~XFEATURE_MASK_SUPERVISOR; + xfeatures_mask_user &= ~XFEATURE_MASK_SYSTEM; cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -242,7 +242,7 @@ void fpu__init_cpu_xstate(void) */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask & (1UL << xfeature)); + return !!(xfeatures_mask_user & BIT_ULL(xfeature)); } /* @@ -272,7 +272,7 @@ static void __init setup_xstate_features(void) cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx); /* - * If an xfeature is supervisor state, the offset + * If an xfeature is system state, the offset * in EBX is invalid. We leave it to -1. */ if (xfeature_is_user(i)) @@ -348,7 +348,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; int i; /* @@ -421,7 +421,8 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = (u64)1 << 63 | xfeatures_mask; + init_fpstate.xsave.header.xcomp_bv = + BIT_ULL(63) | xfeatures_mask_user; /* * Init all the features state with header.xfeatures being 0x0 @@ -440,11 +441,11 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports supervisor states and it uses compacted - * format. Checking a supervisor state's uncompacted offset is + * Only XSAVES supports system states and it uses compacted + * format. Checking a system state's uncompacted offset is * an error. */ - if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { + if (XFEATURE_MASK_SYSTEM & (1 << xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -465,7 +466,7 @@ static int xfeature_size(int xfeature_nr) /* * 'XSAVES' implies two different things: - * 1. saving of supervisor/system state + * 1. saving of system state * 2. using the compacted format * * Use this function when dealing with the compacted format so @@ -480,8 +481,8 @@ int using_compacted_format(void) /* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ int validate_xstate_header(const struct xstate_header *hdr) { - /* No unknown or supervisor features may be set */ - if (hdr->xfeatures & (~xfeatures_mask | XFEATURE_MASK_SUPERVISOR)) + /* No unknown or system features may be set */ + if (hdr->xfeatures & (~xfeatures_mask_user | XFEATURE_MASK_SYSTEM)) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -588,11 +589,11 @@ static void do_extra_xstate_size_checks(void) check_xstate_against_struct(i); /* - * Supervisor state components can be managed only by + * System state components can be managed only by * XSAVES, which is compacted-format only. */ if (!using_compacted_format()) - XSTATE_WARN_ON(xfeature_is_supervisor(i)); + XSTATE_WARN_ON(xfeature_is_system(i)); /* Align from the end of the previous feature */ if (xfeature_is_aligned(i)) @@ -616,7 +617,7 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask. + * Get total size of enabled xstates in XCR0/xfeatures_mask_user. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer @@ -706,7 +707,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { - xfeatures_mask = 0; + xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); } @@ -742,15 +743,15 @@ void __init fpu__init_system_xstate(void) } cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask = eax + ((u64)edx << 32); + xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue * booting without it. This is too early to BUG(). */ - pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask); + pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask_user); goto out_disable; } @@ -759,10 +760,10 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask &= ~BIT(i); + xfeatures_mask_user &= ~BIT_ULL(i); } - xfeatures_mask &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -772,9 +773,10 @@ void __init fpu__init_system_xstate(void) /* * Update info used for ptrace frames; use standard-format size and no - * supervisor xstates: + * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, + xfeatures_mask_user & ~XFEATURE_MASK_SYSTEM); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -782,7 +784,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask, + xfeatures_mask_user, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -801,7 +803,7 @@ void fpu__resume_cpu(void) * Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has(X86_FEATURE_XSAVE)) - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -853,7 +855,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) * have not enabled. Remember that pcntxt_mask is * what we write to the XCR0 register. */ - WARN_ONCE(!(xfeatures_mask & xstate_feature), + WARN_ONCE(!(xfeatures_mask_user & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -1003,7 +1005,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= ~XFEATURE_MASK_SYSTEM; /* * Copy xregs_state->header: @@ -1087,7 +1089,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= ~XFEATURE_MASK_SYSTEM; /* * Copy xregs_state->header: @@ -1180,7 +1182,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; /* * Add back in the features that came in from userspace: @@ -1236,7 +1238,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; /* * Add back in the features that came in from userspace: From patchwork Thu Aug 30 14:38:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581831 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4ECE14E1 for ; Thu, 30 Aug 2018 14:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3D3A2B560 for ; Thu, 30 Aug 2018 14:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B7E262BC25; Thu, 30 Aug 2018 14:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A24622B560 for ; Thu, 30 Aug 2018 14:43:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 522D06B521D; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 401986B521B; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2041D6B521F; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id C84F46B521B for ; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id h4-v6so4029099pls.17 for ; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9YKfHw/0dgglNTt5KC6AtpZS7C4X66pTo/aEPMbL4Vc=; b=U31lFVR2POeKtqjvMNoTWxjrHAr1P+elhaHIlYNFa2MZteo5YRvl8QDYgrZLDRC80j OSbfUuIbRwC3LqIaTXv1SqMFNJ41CohYsyjYsLZa6CJTlYzyDr77Xa5C5ncjm39w+YAA iNOFH2xE4giM7d5Lm2gkvrZkalin4LYnMI2t8XY1ReAvRO9AQw6Q8Rkn2isk9gHXnZXa JhMAuwqRbP1IofPX8AMZm5/67VYchkHPmzyOmJFrZpquDReqBZgy7XYg3v+NIClzzwdR bqi2OcbUfgm/mTfg3kuX2STt7ef0HN0Xcg1u4LJgtK5+qwuy3OWGBPt+SsBP2Ul3Za3v G0Nw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CE66bIHq+OodFDjVXBTpOBNknfWM4MgzkUXBI/cJ1WPSYBCSdG icFaM9284pNNjibhc90etRwhW+UhDKO8J0Cpg67vjSPA+/YUA8CorcRadrcqW2Jl+sFcWA4ETdk WznkKOgjj4wFzFGHTsrME0aOwsvXpvAYu13X4ZMtFWo+ysXr7XkPAtzyb5NMp2rKY9Q== X-Received: by 2002:a17:902:6806:: with SMTP id h6-v6mr389067plk.304.1535640225457; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaoKtS4zZACRl1XlJJqUXSmMVXiHULYUrMhINKB5Z+DqUtENewv8X8Vz6brhI20nA3pP6xJ X-Received: by 2002:a17:902:6806:: with SMTP id h6-v6mr389008plk.304.1535640224269; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=SHsg/hXBbFAFkgpmhkY4If7M2Adq6NfMGijxOYjjvcdR2uq/WoDSZlP8GIBG7rtzNY +7SBQZS9JvqwC/UXMyzeTPPxmCBDkjG4io8yw61Qh4/yXNFiETYR4VoxjsW+nUUC4R77 3wOnlgFv9WqvNLSAydXiSyL9ZWUOjbOfmJbM40di8vjHCubSFzyzY1mXXRmWibrooa4B Awlu2cN0oa53UrFUo5vh3I4QfN2E/79fywRwfg1aE3dq8eJhelkw7X6wL3bSFjenksZj wDt2ZUDY+d9xWdGb8r+JJ2fG2qGMdQI2HKzWfozgAPLpgXxl0iNX63yMVX26FTgu3Jmo Iy9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=9YKfHw/0dgglNTt5KC6AtpZS7C4X66pTo/aEPMbL4Vc=; b=aJglesljiEP1LKTK7L+nF6jT8k1vpINEgo8DUIjVkChQ/iNBuSanrgh085oUJFxlqh LkTMV45Mn0B79B1g+H2s7AelLBRjomyFNheOwSnDeXyZEbzYGC5qJ7BoEsp7ow+PxYdT N4WiEgdCPWFoDhqw7AoM2K3zEMDABa9OpkqEcsp8C+uGEI901SGwFI/weCKEuANBsCbE XsEnt8aq0kzDBEIEa9Q9U6ZSRScMiQBtdjPNoFQvq0KEF9AzoqKG3LBpy4XXVM8IkHOq OvuZnlB1d3jrfTfwSd3E8de3KvFdcv2hSGmwAifQTZpRKwODaI4PMv7HDVWggz/6T3XC RmAw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186668" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 03/24] x86/fpu/xstate: Enable XSAVES system states Date: Thu, 30 Aug 2018 07:38:43 -0700 Message-Id: <20180830143904.3168-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP XSAVES saves both system and user states. The Linux kernel currently does not save/restore any system states. This patch creates the framework for supporting system states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 3 +- arch/x86/include/asm/fpu/xstate.h | 9 ++- arch/x86/kernel/fpu/core.c | 7 +- arch/x86/kernel/fpu/init.c | 10 --- arch/x86/kernel/fpu/xstate.c | 112 +++++++++++++++++----------- 5 files changed, 80 insertions(+), 61 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index f1f9bf91a0ab..1f447865db3a 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -45,7 +45,6 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system(struct cpuinfo_x86 *c); extern void fpu__init_check_bugs(void); extern void fpu__resume_cpu(void); -extern u64 fpu__get_supported_xfeatures_mask(void); /* * Debugging facility: @@ -94,7 +93,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * trigger #GP: */ xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | - xfeatures_mask_user; + xfeatures_mask_all; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 9b382e5157ed..a32dc5f8c963 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,10 +19,10 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* System features */ -#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT) - -/* All currently supported features */ +/* + * SUPPORTED_XFEATURES_MASK indicates all features + * implemented in and supported by the kernel. + */ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ XFEATURE_MASK_YMM | \ @@ -40,6 +40,7 @@ #endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 4bd56079048f..9f51b0e1da25 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -365,8 +365,13 @@ void fpu__drop(struct fpu *fpu) */ static inline void copy_init_user_fpstate_to_fpregs(void) { + /* + * Only XSAVES user states are copied. + * System states are preserved. + */ if (use_xsave()) - copy_kernel_to_xregs(&init_fpstate.xsave, -1); + copy_kernel_to_xregs(&init_fpstate.xsave, + xfeatures_mask_user); else if (static_cpu_has(X86_FEATURE_FXSR)) copy_kernel_to_fxregs(&init_fpstate.fxsave); else diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 761c3a5a9e07..eaf9d9d479a5 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -222,16 +222,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* - * Find supported xfeatures based on cpu features and command-line input. - * This must be called after fpu__init_parse_early_param() is called and - * xfeatures_mask is enumerated. - */ -u64 __init fpu__get_supported_xfeatures_mask(void) -{ - return SUPPORTED_XFEATURES_MASK; -} - /* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch(void) { diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 19f8df54c72a..dd2c561c4544 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -51,13 +51,16 @@ static short xsave_cpuid_features[] __initdata = { }; /* - * Mask of xstate features supported by the CPU and the kernel: + * Mask of xstate features supported by the CPU and the kernel. + * This is the result from CPUID query, SUPPORTED_XFEATURES_MASK, + * and boot_cpu_has(). */ u64 xfeatures_mask_user __read_mostly; +u64 xfeatures_mask_all __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_all)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +85,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -164,7 +167,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) + if ((xfeatures & xfeatures_mask_all) == xfeatures_mask_all) return; /* @@ -219,30 +222,31 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_all) return; + + cr4_set_bits(X86_CR4_OSXSAVE); + /* - * Make it clear that XSAVES system states are not yet - * implemented should anyone expect it to work by changing - * bits in XFEATURE_MASK_* macros and XCR0. + * XCR_XFEATURE_ENABLED_MASK sets the features that are managed + * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be + * set here. */ - WARN_ONCE((xfeatures_mask_user & XFEATURE_MASK_SYSTEM), - "x86/fpu: XSAVES system states are not yet implemented.\n"); + xsetbv(XCR_XFEATURE_ENABLED_MASK, + xfeatures_mask_user); - xfeatures_mask_user &= ~XFEATURE_MASK_SYSTEM; - - cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + /* + * MSR_IA32_XSS sets which XSAVES system states to be managed by + * XSAVES. Only XSAVES system states can be set here. + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, + xfeatures_mask_all & ~xfeatures_mask_user); } -/* - * Note that in the future we will likely need a pair of - * functions here: one for user xstates and the other for - * system xstates. For now, they are the same. - */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask_user & BIT_ULL(xfeature)); + return !!(xfeatures_mask_all & BIT_ULL(xfeature)); } /* @@ -348,7 +352,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_all)*8]; int i; /* @@ -422,7 +426,7 @@ static void __init setup_init_fpu_buf(void) if (boot_cpu_has(X86_FEATURE_XSAVES)) init_fpstate.xsave.header.xcomp_bv = - BIT_ULL(63) | xfeatures_mask_user; + BIT_ULL(63) | xfeatures_mask_all; /* * Init all the features state with header.xfeatures being 0x0 @@ -441,11 +445,10 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports system states and it uses compacted - * format. Checking a system state's uncompacted offset is - * an error. + * Checking a system or unsupported state's uncompacted offset + * is an error. */ - if (XFEATURE_MASK_SYSTEM & (1 << xfeature_nr)) { + if (~xfeatures_mask_user & BIT_ULL(xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -482,7 +485,7 @@ int using_compacted_format(void) int validate_xstate_header(const struct xstate_header *hdr) { /* No unknown or system features may be set */ - if (hdr->xfeatures & (~xfeatures_mask_user | XFEATURE_MASK_SYSTEM)) + if (hdr->xfeatures & ~xfeatures_mask_user) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -617,15 +620,12 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask_user. + * Get total size of enabled xstates in XCR0 | IA32_XSS. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer * that we use 'XSAVES' on, we could potentially overflow the * buffer because 'XSAVES' saves system states too. - * - * Note that we do not currently set any bits on IA32_XSS so - * 'XCR0 | IA32_XSS == XCR0' for now. */ static unsigned int __init get_xsaves_size(void) { @@ -707,6 +707,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { + xfeatures_mask_all = 0; xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); @@ -722,6 +723,8 @@ void __init fpu__init_system_xstate(void) static int on_boot_cpu __initdata = 1; int err; int i; + u64 cpu_user_xfeatures_mask; + u64 cpu_system_xfeatures_mask; WARN_ON_FPU(!on_boot_cpu); on_boot_cpu = 0; @@ -742,10 +745,24 @@ void __init fpu__init_system_xstate(void) return; } + /* + * Find user states supported by the processor. + * Only these bits can be set in XCR0. + */ cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask_user = eax + ((u64)edx << 32); + cpu_user_xfeatures_mask = eax + ((u64)edx << 32); + + /* + * Find system states supported by the processor. + * Only these bits can be set in IA32_XSS MSR. + */ + cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx); + cpu_system_xfeatures_mask = ecx + ((u64)edx << 32); - if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + xfeatures_mask_all = cpu_user_xfeatures_mask | + cpu_system_xfeatures_mask; + + if ((xfeatures_mask_all & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -760,10 +777,11 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask_user &= ~BIT_ULL(i); + xfeatures_mask_all &= ~BIT_ULL(i); } - xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_all &= SUPPORTED_XFEATURES_MASK; + xfeatures_mask_user = xfeatures_mask_all & cpu_user_xfeatures_mask; /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -775,8 +793,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, - xfeatures_mask_user & ~XFEATURE_MASK_SYSTEM); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -784,7 +801,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask_user, + xfeatures_mask_all, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -804,6 +821,13 @@ void fpu__resume_cpu(void) */ if (boot_cpu_has(X86_FEATURE_XSAVE)) xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * Restore IA32_XSS + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, + xfeatures_mask_all & ~xfeatures_mask_user); } /* @@ -853,9 +877,9 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) /* * We should not ever be requesting features that we * have not enabled. Remember that pcntxt_mask is - * what we write to the XCR0 register. + * what we write to the XCR0 | IA32_XSS registers. */ - WARN_ONCE(!(xfeatures_mask_user & xstate_feature), + WARN_ONCE(!(xfeatures_mask_all & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -1005,7 +1029,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SYSTEM; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1089,7 +1113,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SYSTEM; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1182,7 +1206,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: @@ -1238,7 +1262,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: From patchwork Thu Aug 30 14:38:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581833 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7CA2514E1 for ; Thu, 30 Aug 2018 14:43:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BD8D2B560 for ; Thu, 30 Aug 2018 14:43:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FAF22BC83; Thu, 30 Aug 2018 14:43:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF2852B560 for ; Thu, 30 Aug 2018 14:43:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AA556B521B; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 511DD6B5220; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35FE26B521D; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id E2F056B521E for ; Thu, 30 Aug 2018 10:43:45 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id p5-v6so4852508pfh.11 for ; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=HYv72t+ibUOa7b6wsMKk0mmqvwO5mzmA3sLIsJC1ojE=; b=l54y48kSXGo8ZgV/8ySjhTMDUCMx4LMZFQUktHym2IBHATS/nBtAyxrKyNH2CAk//F J8Ug6pgT/xFqgMyME3qPx6b6+1i4Zkge3HSlWzRvjwSbFQHITev2Jv+NqAEDh9DHMF7Z h5Veyl9f3znCjo2PT/0Blca+e2QAx4tQzL7IvPJuJHnxs28smo1L4tSi1FewCsuM2lMD R4IErKPR2ZL3aJlRRUrEPt+kMWEH8f8LEmg03MVpK7pz0+RE3SVHvR4Bgwolp9KIJv/I 11u8xmQ32v3W/Q9dG5BoHH0yIyMJTqgQcOtygFoetIPmP6/hhmgd9/iRT8TN3Zqy71mS b93w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DsfzYJBvGi9BCYJkV+0VgYpc2NQKlBYgpawXG6rcx7L272dpuw m7xcfhRicEoCnjhiT3etLQqP2CLjmiJ+kWERWd2ByulK1wdaP8Rfw550dyjqGF6z2cgPj1mIpim ITV6Y+WpxlwldFNCTUcxD3+JXBgUEse7HWgOYsV/bTH5ghAD4DuHe7mg18EmDux8Z8g== X-Received: by 2002:a62:868b:: with SMTP id x133-v6mr7858015pfd.252.1535640225604; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaBQgPj6VJ/zfm4vSs0wF9Ei+ewxTaoEyvGowsC1Xw6OBjPdA3p7+YbB0QeaXNsgTNXJWxa X-Received: by 2002:a62:868b:: with SMTP id x133-v6mr7857964pfd.252.1535640224630; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=R2vYu0RaNbken3WYuh6azU1vE4/iAfb0Bw6D1B2Ib7T0M/hE9GXw2opTwUPtVS/XRm JT11BSSgFqKKU3FHibd8SXRhp/lGBiTquMdkn8E8puWapvX7jbRaeFhgXUfa26AJHkWp 7O8vLLsQGYdvjbUAlkkQg5cI1xO0xBGIGLMO90zm2SGkkJrnJCtf+v9A9ZWjXn9Qo7Ie JfZgeLVPqnICfYgqQHouNDQOcbGrpGIeCHQdExgjHA+L8NDMd18bwipMO525dMvnabRm y0LaT5BuFdKXY7q/qP+PsmV1+vSXe+0M1T1YPLe7ef3tY5a7HRw0LLQE64VymRQHOSPc 0SRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=HYv72t+ibUOa7b6wsMKk0mmqvwO5mzmA3sLIsJC1ojE=; b=P7+HTblB5cuAKCcNbkuK+bnpItWA7NUS4AiS1qER66N/MDdnVaEoukrix83KRcV8xt P3F9hPOieSG3zYP3gS62T0KMQSRYJ6hiNrGOCeALkLpcevnfTf9xfvV53YM53VhPSDFn yPenuaq6rv2e38MowZL5FI5s716Tn8N/TvROO5rLIGHVClFne/jMrd5oK53i9hxIktkY +BxgXCsMJpKPWccdQHgdp8YCeB1Mu5MBCDNdFSk2/wbC0mWTSqZIE+WmeSiQKahj8gEm PJr7V6aGfTL727k9yXyVcqYgxax/gM1jUonOGHKr2gBVSvg726G9Rx/E8nINE3dkWvp2 nYrg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186671" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 04/24] x86/fpu/xstate: Add XSAVES system states for shadow stack Date: Thu, 30 Aug 2018 07:38:44 -0700 Message-Id: <20180830143904.3168-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Intel Control-flow Enforcement Technology (CET) introduces the following MSRs into the XSAVES system states. IA32_U_CET (user-mode CET settings), IA32_PL3_SSP (user-mode shadow stack), IA32_PL0_SSP (kernel-mode shadow stack), IA32_PL1_SSP (ring-1 shadow stack), IA32_PL2_SSP (ring-2 shadow stack). Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index 202c53918ecf..e55d51d172f1 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM, XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, + XFEATURE_RESERVED, + XFEATURE_SHSTK_USER, + XFEATURE_SHSTK_KERNEL, XFEATURE_MAX, }; @@ -128,6 +131,8 @@ enum xfeature { #define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM) #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_SHSTK_USER (1 << XFEATURE_SHSTK_USER) +#define XFEATURE_MASK_SHSTK_KERNEL (1 << XFEATURE_SHSTK_KERNEL) #define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE) #define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \ @@ -229,6 +234,23 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control flow Enforcement user states + */ +struct cet_user_state { + u64 u_cet; /* user control flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +} __packed; + +/* + * State component 12 is Control flow Enforcement kernel states + */ +struct cet_kernel_state { + u64 kernel_ssp; /* kernel shadow stack */ + u64 pl1_ssp; /* ring-1 shadow stack */ + u64 pl2_ssp; /* ring-2 shadow stack */ +} __packed; + struct xstate_header { u64 xfeatures; u64 xcomp_bv; diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index a32dc5f8c963..662562cbafe9 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -31,7 +31,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_SHSTK_USER | \ + XFEATURE_MASK_SHSTK_KERNEL) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..25311ec4b731 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -130,6 +130,8 @@ #define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT) #define X86_CR4_PKE_BIT 22 /* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23 /* enable Control flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) /* * x86-64 Task Priority Register, CR8 diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index dd2c561c4544..91c0f665567b 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -35,6 +35,9 @@ static const char *xfeature_names[] = "Processor Trace (unused)" , "Protection Keys User registers", "unknown xstate feature" , + "Control flow User registers" , + "Control flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = { @@ -48,6 +51,9 @@ static short xsave_cpuid_features[] __initdata = { X86_FEATURE_AVX512F, X86_FEATURE_INTEL_PT, X86_FEATURE_PKU, + 0, /* Unused */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_KERNEL */ }; /* @@ -316,6 +322,8 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_ZMM_Hi256); print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); + print_xstate_feature(XFEATURE_MASK_SHSTK_USER); + print_xstate_feature(XFEATURE_MASK_SHSTK_KERNEL); } /* @@ -562,6 +570,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_KERNEL, struct cet_kernel_state); /* * Make *SURE* to add any feature numbers in below if From patchwork Thu Aug 30 14:38:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B576E5A4 for ; Thu, 30 Aug 2018 14:46:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A31712B560 for ; Thu, 30 Aug 2018 14:46:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96B5C2BC83; Thu, 30 Aug 2018 14:46:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A920C2B560 for ; Thu, 30 Aug 2018 14:46:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27A156B524D; Thu, 30 Aug 2018 10:45:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1B4E76B524F; Thu, 30 Aug 2018 10:45:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 054566B5250; Thu, 30 Aug 2018 10:45:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id AC78C6B524D for ; Thu, 30 Aug 2018 10:45:10 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id 90-v6so4016190pla.18 for ; Thu, 30 Aug 2018 07:45:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=AxqOQSs5/Dk8NzVpOg+9BUGegWkgyMooFhfPxwwmKMM=; b=aAoTx77Nw8jXaLrie6EKz7d59JQITozmvjQj+1vrLjuxC+1wjDqF827Jq8Ak2s/vxi x0OzumcNCKOPtQ1mD1VwP+sXZ4x47gskqn4NqnPzp+quJzNDUAh4sDeI+8GEl+oMLnhR xuG382ckhQgmg8SDaugv9P0Z5uIRHdHgTABkXUucg04yjC2pdV5xVdHhCDI8KM9hxHXH 2hn7xrxy67cRea91ELtVSBOH9w/p6AXsTMVhCtSBTSfHvZO82n/7D+bDcnaQh5VWi4q6 4Grjb/BxxhRRuvIzUPIliqJjHCm8RcaqEBKwVbwmJM6r7b1RuEbQj1kajsmlPxJ26RRA p4+w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BHB42Yx3GV0oUzauBzUgRmPK6WyCrH+1nIu/IRdnwMQBB0/D1F 57tsaEL6HGzLM2q3eKDXt1Ga4ee1IB0488zcx0MpJ5q2/AY4NlCMJ/h6wyM+6BAYOGGUN13ngfT EMSPDpd0vDtb5dxlKJfgHmJ/2Fu5w/RR9re23nPkFFmbrwsVk49btz5RXTCWBM7/u3A== X-Received: by 2002:a17:902:d881:: with SMTP id b1-v6mr10543715plz.191.1535640310363; Thu, 30 Aug 2018 07:45:10 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbUnCwOMerSehKNWkNKFrxvDTITeP6FIvA4LjJdHi2zEpZhXQFMtGoTlK7fCKeChTqqKO1I X-Received: by 2002:a17:902:d881:: with SMTP id b1-v6mr10539453plz.191.1535640224842; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=CbLksHS7l0F78/oTR799Le3UEyAdyMlK16IwmJ3GHMJpMZNdMv1Kpm9toMWqs5i5Mu MgMvVSuYLlcoFnqcE2Y/B6GAiGd/8XqHitOXFu0OW2s6dqLE9dTJAQlTN+GlQIeRICQY X7fZFqI6UvcQ9xA5k796i4AflN6TWPj9gKEBZa9p1SYml7ijsAU/AhmSN3+2d/fNgKSt IVr8jYI3uZdIYAVjBdJSvt8MI3hFpTfKjt5efTAaAnxTNeAfRhALJpW4/Xi42dAYbZpX DWez28dQmlj6jeuWSnzNmUN7FveIQXmDDeaNBsDv1xvvYBPTbAwyfKK+AwPOXW79JlSn mssA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=AxqOQSs5/Dk8NzVpOg+9BUGegWkgyMooFhfPxwwmKMM=; b=f2FTLXAlHx2p26TASJFDhT5EpV8Na79rB0iC6kDJFB5Oqbu0qmWNt/vk6oR4y9bx4C z29Dw61WubTogmmkLiZstEloD5CyY0I2GA8IiSePZzuA3FanZYEnqI6L28h79RSI1Sgp uipdkJeVT6HLQGofzpz9gi0Ne3f+Mgmfin0b0/5rmSoWFmpIiaIjOvg2TXrVHCYC9ozi al+1ZuNwugEK0lg2dsdaVw6+AhxIRZrioqAaPux2xcads4ZdoMA4LEmHDudNN2T8pPxH w4EVCeqHimzTbW/5/qTWytqtBt359zXjDNq8Gb27Xue+gdkS3Ice3UjkjNgIC/BDJveI hEqQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186675" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 05/24] Documentation/x86: Add CET description Date: Thu, 30 Aug 2018 07:38:45 -0700 Message-Id: <20180830143904.3168-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Explain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters. Signed-off-by: Yu-cheng Yu --- .../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 11 + Documentation/x86/intel_cet.rst | 252 ++++++++++++++++++ 4 files changed, 270 insertions(+) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9871e649ffef..b090787188b4 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. diff --git a/Documentation/index.rst b/Documentation/index.rst index 5db7e87c7cb1..1cdc139adb40 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -104,6 +104,7 @@ implementation. :maxdepth: 2 sh/index + x86/index Filesystem Documentation ------------------------ diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst new file mode 100644 index 000000000000..9c34d8cbc8f0 --- /dev/null +++ b/Documentation/x86/index.rst @@ -0,0 +1,11 @@ +======================= +X86 Documentation +======================= + +Control Flow Enforcement +======================== + +.. toctree:: + :maxdepth: 1 + + intel_cet diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst new file mode 100644 index 000000000000..337baa1f6980 --- /dev/null +++ b/Documentation/x86/intel_cet.rst @@ -0,0 +1,252 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protection against +return/jump-oriented programing (ROP) attacks. It can be implemented +to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented for the 64-bit kernel. +Thirty-two bit applications are supported under the compatibility +mode. + +CET includes shadow stack (SHSTK) and indirect branch tracking (IBT) +and they are enabled from two kernel configuration options: + + INTEL_X86_SHADOW_STACK_USER, and + INTEL_X86_BRANCH_TRACKING_USER. + +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later +are required. To build a CET-enabled application, GLIBC v2.28 or +later is also required. + +There are two command-line options for disabling CET features: + + no_cet_shstk - disables SHSTK, and + no_cet_ibt - disables IBT. + +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT. + +[2] CET assembly instructions +============================= + +RDSSP %r + Read the SHSTK pointer into %r. + +INCSSP %r + Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated + in the operand register. The GLIBC longjmp uses INCSSP to unwind + the SHSTK until that matches the program stack. When it is + necessary to unwind beyond 255 steps, longjmp divides and repeats + the process. + +RSTORSSP (%r) + Switch to the SHSTK indicated in the 'restore token' pointed by + the operand register and replace the 'restore token' with a new + token to be saved (with SAVEPREVSSP) for the outgoing SHSTK. + +:: + + Before RSTORSSP + + Incoming SHSTK Current/Outgoing SHSTK + + |----------------------| |----------------------| + addr=x | | ssp-> | | + |----------------------| |----------------------| + (%r)-> | rstor_token=(x|Lg) | addr=y-8 | | + |----------------------| |----------------------| + + After RSTORSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | | + |----------------------| |----------------------| + + note: + 1. Only valid addresses and restore tokens can be on the + user-mode SHSTK. + 2. A token is always of type u64 and must align to u64. + 3. The incoming SHSTK pointer in a rstor_token must point to + immediately above the token. + 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK. + 5. 'Bz' is bit[1] of a rstor_token indicating the token is to + be used only for the next SAVEPREVSSP and invalid for the + RSTORSSP. + +SAVEPREVSSP + Store the SHSTK 'restore token' pointed by + (current_SHSTK_pointer + 8). + +:: + + After SAVEPREVSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) | + |----------------------| |----------------------| + +WRUSS %r0, (%r1) + Write the value in %r0 to the SHSTK address pointed by (%r1). + This is a kernel-mode only instruction. + +ENDBR + The compiler inserts an ENDBR at all valid branch targets. Any + CALL/JMP to a target without an ENDBR triggers a control + protection fault. + +[3] Application Enabling +======================== + +An application's CET capability is marked in its ELF header and can +be verified from the following command output, in the +NT_GNU_PROPERTY_TYPE_0 field: + + readelf -n + +If an application supports CET and is statically linked, it will run +with CET protection. If the application needs any shared libraries, +the loader checks all dependencies and enables CET only when all +requirements are met. + +[4] Legacy Libraries +==================== + +GLIBC provides a few tunables for backward compatibility. + +GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT + Turn off SHSTK/IBT for the current shell. + +GLIBC_TUNABLES=glibc.tune.x86_shstk= + This controls how dlopen() handles SHSTK legacy libraries: + on: continue with SHSTK enabled; + permissive: continue with SHSTK off. + +[5] CET system calls +==================== + +The following arch_prctl() system calls are added for CET: + +arch_prctl(ARCH_CET_STATUS, unsigned long *addr) + Return CET feature status. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = SHSTK/IBT status + *(addr + 1) = SHSTK base address + *(addr + 2) = SHSTK size + +arch_prctl(ARCH_CET_DISABLE, unsigned long features) + Disable SHSTK and/or IBT specified in 'features'. Return -EPERM + if CET is locked. + +arch_prctl(ARCH_CET_LOCK) + Lock in CET feature. + +arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr) + Allocate a new SHSTK. + + The parameter 'addr' is a pointer to a user buffer and indicates + the desired SHSTK size to allocate. On returning to the caller + the buffer contains the address of the new SHSTK. + +arch_prctl(ARCH_CET_LEGACY_BITMAP, unsigned long *addr) + Allocate an IBT legacy code bitmap if the current task does not + have one. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = IBT bitmap base address + *(addr + 1) = IBT bitmap size + +[6] The implementation of the SHSTK +=================================== + +SHSTK size +---------- + +A task's SHSTK is allocated from memory to a fixed size of +RLIMIT_STACK. + +Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, we can estimate a large +enough SHSTK to cover the condition that both the program stack and +the sigaltstack run out. + +The kernel creates a restore token at the SHSTK restoring address and +verifies that token when restoring from the signal handler. + +Fork +---- + +The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be +read-only and dirty. When a SHSTK PTE is not present, RO, and dirty, +a SHSTK access triggers a page fault with an additional SHSTK bit set +in the page fault error code. + +When a task forks a child, its SHSTK PTEs are copied and both the +parent's and the child's SHSTK PTEs are cleared of the dirty bit. +Upon the next SHSTK access, the resulting SHSTK page fault is handled +by page copy/re-use. + +When a pthread child is created, the kernel allocates a new SHSTK for +the new thread. + +Setjmp/Longjmp +-------------- + +Longjmp unwinds SHSTK until it matches the program stack. + +Ucontext +-------- + +In GLIBC, getcontext/setcontext is implemented in similar way as +setjmp/longjmp. + +When makecontext creates a new ucontext, a new SHSTK is allocated for +that context with ARCH_CET_ALLOC_SHSTK the syscall. The kernel +creates a restore token at the top of the new SHSTK and the user-mode +code switches to the new SHSTK with the RSTORSSP instruction. + +[7] The management of read-only & dirty PTEs for SHSTK +====================================================== + +A RO and dirty PTE exists in the following cases: + +(a) A page is modified and then shared with a fork()'ed child; +(b) A R/O page that has been COW'ed; +(c) A SHSTK page. + +The processor only checks the dirty bit for (c). To prevent the use +of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as +DIRTY_SW for (a) and (b) above. This results to the following PTE +settings: + +Modified PTE: (R/W + DIRTY_HW) +Modified and shared PTE: (R/O + DIRTY_SW) +R/O PTE, COW'ed: (R/O + DIRTY_SW) +SHSTK PTE: (R/O + DIRTY_HW) +SHSTK PTE, COW'ed: (R/O + DIRTY_HW) +SHSTK PTE, shared: (R/O + DIRTY_SW) + +Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs. + +[8] The implementation of IBT +============================= + +The kernel provides IBT support in mmap() of the legacy code bit map. +However, the management of the bitmap is done in the GLIBC or the +application. From patchwork Thu Aug 30 14:38:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581837 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DA4C14E1 for ; Thu, 30 Aug 2018 14:44:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CB1F2B560 for ; Thu, 30 Aug 2018 14:44:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 70B5F2BC25; Thu, 30 Aug 2018 14:44:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC5862B560 for ; Thu, 30 Aug 2018 14:44:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 512336B5224; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 11AF36B5223; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0B96B5228; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 45FAE6B521F for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id w19-v6so4866656pfa.14 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=DEXaEudwtoTcbEpnYbJ0V54+eLZ+SRMKFWLcfqIPsGo=; b=fDIrvKDkHwCtCcRLPvNyiBaM4EfGSaxx0mIM8q7JrvImCF9IM/WQHVZvFEyvSWFj9U p3z9SpLRbnELWn3PKEqlRUtocyOA/DxrA0MKJO52ckn0KWOFewCi8ymcOdTxF7kfNc9z jINWLPibsqaFxAFNk4q7Ui32vmQZsn2Fy+lr6eybmteHyz8FcuYnC55Oe/EIo4VvO7Mt 9Mjn1pUsgm6b6ptGUD0ZB6O3TtDFUL604wkGk4KKRWfF6mlqq8UZ1WZ+7GGAlnHxqGBR lENf8lGOpQOjnuCaC31WzQmJrSppM5fvojEXjL022hksW/Soy1UOseTfohA+542JNMXR Ed9A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BCvk3LYr88FQUg/GqHRtQxBOIgYknxO1oW6WX0rOFcHGdbMI/h QuDS7oROeu5nKwR8cWh8i4qsSBJQ8JRHu6Ks+8RIpPtxJjWcNmWJq99o2dIi+dItXBJmg24+Nn1 a+koNoIlGwR7J6IF2gQ9Ik69bEeAzg+o//GMzvZ1G3ZqJzP8K5HSLMDweYk4I5bnPSw== X-Received: by 2002:a62:6c42:: with SMTP id h63-v6mr2055156pfc.65.1535640225958; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZqcXzGEyKq55f5hJsJHB//UAHb/Pm5s7QnoEbeHfqBa2RuldKsTHJjAQYg3lFEa1Aq//gt X-Received: by 2002:a62:6c42:: with SMTP id h63-v6mr2055115pfc.65.1535640225026; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=p7tBUrM5N2Ti248eX8HPa9w6pDztYk0a4+3nh/uEIHZREv59WGNih/KU6X0rkPLI/k VOOmj4ANMpPE3PjDqgKwkjYZhgsyqOIeWSN3/TC6YzKCyAB+dAKcYqfGV8BphI2aTh3E V8xKXgcMJ2/5eXSs6i0vYhPQrgEJtNO0uFy4kvayO69wthKSWYZfV6QZ0+DCrd4Xatzh QBg7JFcZfAngweNfnaGMgLF1ceNhwHYJtpP+t5lufeZFc4T0xFqD0soXVNM3cPq06UJi 3e0jJLcSpEZHGqWfq4nim1f+SFgu5F8uWm9ILt8cgriEcMi7n/Zy9UU8oOm3PRPNh1PK kqJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=DEXaEudwtoTcbEpnYbJ0V54+eLZ+SRMKFWLcfqIPsGo=; b=bnm3RFqAgBhF51SEt+++ctuvnCpyiXLZjPLEXKnOABYR+mpbe0KWHZGoo1KZa68C31 gmDQB5rShgxOB3Ta8nKPPMeFcxl0/NjmLl/tGxfCsQbghcvZwYZJv4iKeFEkhcl3DZK9 X9VHwoA+lSTAJLQJj6Ru/APGVsE2ZABoeN0NpPlRsMooC8IFnwDxx63kDnuXObtEBQgR 1y5IoPzKSsHDB+uCY6/kXpsNqSi+PKoaR/To/slIbDNrgOrvM15pwEtXxDLRIgvlxGy8 pa4jeF0CUda1acY6ThExgrQO1eRQ/oVkYB5FxvyqoGqZvjuJx9QV3n766hj54IHFXemL Eq7A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186679" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 06/24] x86/cet: Control protection exception handler Date: Thu, 30 Aug 2018 07:38:46 -0700 Message-Id: <20180830143904.3168-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction. The control protection exception handler works in a similar way as the general protection fault handler. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/traps.c | 58 ++++++++++++++++++++++++++++++++++++ 4 files changed, 66 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 957dfb693ecc..5f4914e988df 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1000,7 +1000,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 3de69330e6c5..5196050ff3d5 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void); @@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long); #endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *, long); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long); @@ -142,6 +144,7 @@ enum { X86_TRAP_AC, /* 17, Alignment Check */ X86_TRAP_MC, /* 18, Machine Check */ X86_TRAP_XF, /* 19, SIMD Floating-Point Exception */ + X86_TRAP_CP = 21, /* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ }; diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 01adea278a71..2d02fdd599a2 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = { #elif defined(CONFIG_X86_32) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32), #endif + +#ifdef CONFIG_X86_INTEL_CET + INTG(X86_TRAP_CP, control_protection), +#endif }; /* diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index e6db475164ed..21a713b96148 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -578,6 +578,64 @@ do_general_protection(struct pt_regs *regs, long error_code) } NOKPROBE_SYMBOL(do_general_protection); +static const char *control_protection_err[] = +{ + "unknown", + "near-ret", + "far-ret/iret", + "endbranch", + "rstorssp", + "setssbsy", +}; + +/* + * When a control protection exception occurs, send a signal + * to the responsible application. Currently, control + * protection is only enabled for the user mode. This + * exception should not come from the kernel mode. + */ +dotraplinkage void +do_control_protection(struct pt_regs *regs, long error_code) +{ + struct task_struct *tsk; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); + if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code, X86_TRAP_CP, SIGSEGV) == NOTIFY_STOP) + return; + cond_local_irq_enable(regs); + + if (!user_mode(regs)) + die("kernel control protection fault", regs, error_code); + + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control " + "protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + printk_ratelimit()) { + unsigned int max_err; + + max_err = ARRAY_SIZE(control_protection_err) - 1; + if ((error_code < 0) || (error_code > max_err)) + error_code = 0; + pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); + print_vma_addr(" in ", regs->ip); + pr_cont("\n"); + } + + force_sig_info(SIGSEGV, SEND_SIG_PRIV, tsk); +} +NOKPROBE_SYMBOL(do_control_protection); + dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) { #ifdef CONFIG_DYNAMIC_FTRACE From patchwork Thu Aug 30 14:38:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D387E5A4 for ; Thu, 30 Aug 2018 14:44:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C25FF2B560 for ; Thu, 30 Aug 2018 14:44:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B5B0D2BC83; Thu, 30 Aug 2018 14:44:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 483B12B560 for ; Thu, 30 Aug 2018 14:44:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F32D16B5221; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 98ED66B5225; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05D956B5227; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 0BB546B522B for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id f32-v6so4191679pgm.14 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=k4ilXAgtM4WeGMdX7a31F8vTlTq5Mx7cW6GXmMEayEc=; b=OhqMop5j2GLVn9daXSlaCzZBQx/dCNS08y2IlmBYBSvr/CKsG1YjVAkB+WCUw/cVE0 CEc0tQYXXpNl4BdaDArPrlHX3JMkv1cV6sX58R17+xM5edMKOFvRKCq/Vix6z95513a4 pk9szWfgB+r6O7FfC2B/ANyC6h0MLpU+ohGwAcbYd7/3KILd0slN6kJsfSHEd4OjleyX 0bPLlMa7LQ7JXAy3rpaHti8Yt8dXarIyUPITAIOlRqh3m5qQ9gnq1WKm+fRLysD78u+q SKk4+eZRRGmir3Ah/bvlVzvT0vT6gWUnr5PPM6HNcR0KxcXlnIF7S/mbsAmxoEYrbLQB 1yuA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BjvETIGxvj83+NKNs82Qr0Cq8FJIbIhKtjfGM5p8cbAli80LQE DOEQK/GofYrFWRCtLu3G+wVADBZ+vPG1m9QuGe64bHEzDQQkVK0k9H+a/wjSCoaHsriwG016cD8 ROD11q4ZWWIEL/yYvBJd0x9719BwH0hB6pMhIQy4vuNNJiCRM1gaFjf1CTOIMxDDbAw== X-Received: by 2002:a62:9541:: with SMTP id p62-v6mr10988200pfd.194.1535640226730; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYHxoESj+KX/ZBcGFGdRNTJ0P31EINj+X1RTbV+gcZrBq2ij14okqMAakFIxl/8RA5ggzNY X-Received: by 2002:a62:9541:: with SMTP id p62-v6mr10988173pfd.194.1535640225919; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=XzEjMbjWMIfvAlNhCFkKvg20gH77PjLPbsnwW/1lpweu8rxs0TMmGshjvZnWaV/yCt RXQLqL6zfobjd+GFbr6BxsMhEQNyLqnmNVIp6i7DJpkd8JYgjDgthAoCzBmEVVwSZKy7 ftI+96GVwrc6d5tE7TwBhN26qLEmRkt9RSAAlZ79G/d4QyygRqOI8JcPNJFwTxTaxmWP 687K12+XYy4TRrQsqwfofh/CCVgC0dcs7S56AuyrqqM7A/V16VIrOhMYFb7tizBnh9xd HyIgWyWHvQNKMxMkmpo3+ye4GU8wXacP5GIos+GLboeqgYW+uraV8GxzOLoxRXEeK+Xu Ye6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=k4ilXAgtM4WeGMdX7a31F8vTlTq5Mx7cW6GXmMEayEc=; b=RmnZZ4K9F+BLlzttXqvjQlSHHO4R1sxCUpM/VIrFPyN9AcsDZsfMC7hzmqAijhqxr4 J/znIeMZ8uzg/z1zlRUx3nwLzP9rciFdzoQE8pSWTJxbhXNKGP+US/LFiBgmCsypXSCU F+154079jlmybNVMpLBE+LUQAUtfujkaISvyHhDuYWOIuAPFiBwPiiYS/MHM/dEc4DUV Nycy7rrpc+9QtGPq6PJMGSH/oYTx5rIeL+pvZSzyBF3VNns+ho7TRvtuwcx1gKwnQvxq QYDnQQ11u38aFh/yyhbqYYjEspInN16bPGO/WiSqeR0nZWEC2CodpBvgdbJDD15nM07G /M4g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186683" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 07/24] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Date: Thu, 30 Aug 2018 07:38:47 -0700 Message-Id: <20180830143904.3168-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER. An application has shadow stack protection when all the following are true: (1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled, (2) The running processor supports the shadow stack, (3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack. If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection. The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 31 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index c5ff296bc5d1..017b3ba70807 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1913,6 +1913,30 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select X86_INTEL_CET + select ARCH_HAS_SHSTK + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 94859241bc3e..00927853e409 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -152,6 +152,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Thu Aug 30 14:38:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581855 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 602FF14E1 for ; Thu, 30 Aug 2018 14:44:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D2382B560 for ; Thu, 30 Aug 2018 14:44:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3F10E2BC83; Thu, 30 Aug 2018 14:44:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D1D42B560 for ; Thu, 30 Aug 2018 14:44:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 503E76B522A; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 27C6B6B5220; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 230AF6B521E; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 02CB16B5229 for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id t4-v6so4066085plo.0 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OC+LuoybN9nY4mL1+dVBYqXduCqRpQbUrL1Rlsz3TUM=; b=CiNLUhyZZt/m9vfD2gFkHChiMRoz/ZgaCdSFTKxtyubC6NNnDGgxTzMjO7oRiCsx25 VIQt9rYZJdecjQA7BTxb5p1UwvEmVk8u/+QYNKKFgtf5lNL7dotj5ZLM4c84pCxnadex UI3rfvTdzh+APHU7yBgdb402V6gisGujyBum8SU8LcWYdT5/6OF+n8meRXD6EZxn3TTv unA4sOKnZgsVBXdHS+K008jxdn/hlmIbh6QPmOPz6q6BYVAexXS+MugQSaRqYqNx0qnk DVv9TzncYnl4a5TSOCZGw6I5Bj0IXMW2EC45SAY6T7t2Rb5qCbMZgOYsaY8dbFjpSgdm qSjA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Dcg8dJOy4y+CPge0puJzpwrXjRkQ/ZHNABpEHxYZJQAvELAd6O 37WZn5Q1syQ2YoeTxeHNMsoDiVL4/ufCi6U38JfYu+NUszBqu+TyAv1ya75sb0m6u05QC/FcHpv gkKywxy1HYaemEije3qCCV2edkIY8toelPsowW02zuC2y5HjNvIUUf/fXLBibaus2iA== X-Received: by 2002:a63:3cc:: with SMTP id 195-v6mr9748548pgd.229.1535640226625; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb8c5C8KLzwLXbPuOwLYzbrk8fppG4j9F0QtezPuOUhVDVzPLQzX5hiskSldXFhizPAECpW X-Received: by 2002:a63:3cc:: with SMTP id 195-v6mr9748498pgd.229.1535640225387; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=QvnuUZWFucu1tWobEBJ9v8nmBOGzlAnxc925XGugeLJZKhglTj2Q24OXcyYK/ytvdi F+YBMhXaoSpo6rnPk6Q++1YJwvg+22VASF6SfihYjwwraUiynnWDHcymVGl9uAyR80XW 9M8FYoP9yrHXI41pW/LtmcKaY/Hf43z7o8xnM6aJza4eez5xZ3ANPDbP81mhCTlLpbJM 64YBPatE3UrvB29ypuPfdIwdpyAptbRpnyN+lxEeGrpp1QItsuN7M7f3TJ+jwQIPt8/7 vFI8FtAUIzBPQVICqPeiWZAFNOpCWa07ZhsbPnWgsHuKtiOfruUEspbti+ghLzPEAwoY FH3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=OC+LuoybN9nY4mL1+dVBYqXduCqRpQbUrL1Rlsz3TUM=; b=ur8QRN7N+mewQg3mJlcDMlMQdDcWcX1dlE5CAYj7les+rdFagr1678ou3o4clKVNi/ HQLxfFTV2yO79NxdnN4RQXTCJaMGWhLWChs3QknWKCO1Ueg0XZBiX618VdtGNxq1VKO9 tWh9HSigbYNEx48vv5Qt5yzBe5aAr6bsBw/0/qOYYn2O26iAJDLd6kdMUWO2dUngoYbS xRT0+lIBN11ROiAHvwRiH0UlxENQdgIeB33+vbdzffp/JpNsCBrtogcfVrQ9VZpOJ/q4 ix7D/POncp0Ec4N4i0uB6xotFE6GygcqGEMLWFESRiCpXLw01ieacpJSLezMM+8z9n2p gm5A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id t2-v6si6809115pge.64.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186704" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 08/24] mm: Introduce VM_SHSTK for shadow stack memory Date: Thu, 30 Aug 2018 07:38:48 -0700 Message-Id: <20180830143904.3168-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP VM_SHSTK indicates a shadow stack memory area. The shadow stack is implemented only for the 64-bit kernel. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 8 ++++++++ mm/internal.h | 8 ++++++++ 2 files changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..f40387ecd920 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -224,11 +224,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -266,6 +268,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_MPX VM_NONE #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif diff --git a/mm/internal.h b/mm/internal.h index 87256ae1bef8..d5ee0c04a6c4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -280,6 +280,14 @@ static inline bool is_data_mapping(vm_flags_t flags) return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE; } +/* + * Shadow stack area + */ +static inline bool is_shstk_mapping(vm_flags_t flags) +{ + return (flags & VM_SHSTK); +} + /* mm/util.c */ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma, struct vm_area_struct *prev, struct rb_node *rb_parent); From patchwork Thu Aug 30 14:38:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61BFB174A for ; Thu, 30 Aug 2018 14:45:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 510BE2B560 for ; Thu, 30 Aug 2018 14:45:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 44F9D2BC83; Thu, 30 Aug 2018 14:45:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F0382B560 for ; Thu, 30 Aug 2018 14:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2F666B522D; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A3F916B522B; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42CC96B522D; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id C7C1A6B5220 for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id f32-v6so4191700pgm.14 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Y/Zb3HhLucBBomchluIKoMonk6jmkwpgsk7jxNgQ7dA=; b=kKeFYkJ/rY6VexdD0WEY+1fByTnRhaLzwuqh+DCGd4QYfde5CmnMyaF9y6L7iv3Gsu SNYyChdO+Ue+7IbHRnANKI3s+drY7Tp10/WL1HEIW7/Y16pHW8ONGnlTxhBaregrdqaA qZ6JcBHgWXzyg3CugHLRn55h0C+OIJPjL5ptbliygN+ZagZvVauNTyNEW3zLHFFflgLe TQvxa3JDmOd4UQaLFVS2w4lVySgw+2tSWZCA+wxQ1s/iDfoGuHpb5Tlf1+QgOhS8UcZV QpGXy3UqvMVWR0BiB5ROvRHGbDQtc0YBXXJJeSTldA0+sWUQUxsGNbNWsKL1kelhGv1o mOfA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AYEPhb3fPozZpvkMmQM+r8hIV+oDTMacyLjiJwPC6NBn6WjdmW YHFMmQbit1hu2WkYcUmZUZAGjSV2TrXOcA2SeF4s62FXqeyOhrkJn3aiL79wLJN+XF/mt24v7xr VLWF8qEsuyDfHRIFa6QikNYnA54By2DSmFHLezNzvSnvKpmZeP2CAQWwj7b59DkRloA== X-Received: by 2002:a17:902:8215:: with SMTP id x21-v6mr10328063pln.175.1535640227499; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbRm5xRNYeovMEL7LNFNXa3UPzgqCpzGwHYtuO7MQr36oikajaUckRIvf2vCWLLB7X/VbgO X-Received: by 2002:a17:902:8215:: with SMTP id x21-v6mr10328022pln.175.1535640226618; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=movyqjud3OB+L5y67XiZ2glhP2in0/wtYO2aFyu0c6i8nvOaoBY3cujdSHk5ji7icx c1XVWwYo/4rAEyWB+E2ahlcBOsV2GQvP9IgMlxEKX/yQ9zvw9jXvCzhtbNIIHKG09gKE 2zLMxWuUNlISPTQdxXsGsA0MFGF+6hQASeUv+loyYkO38of3hum4SsHsfht22hYyZcb5 WM4xIafAp8k90RinDMilvcp9GoPXQcRVA9C0rSwzjiFregwZyex8sqk8XWRfCo2mPJd3 wf1oWUtJDAsU1aAoP1thfSKOB2OVXCLHFOWw8snIqLglBzKDpUUr1N/DMckUNMn/1d0k z05w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=Y/Zb3HhLucBBomchluIKoMonk6jmkwpgsk7jxNgQ7dA=; b=gztnM9IxQQHtGTakQoiI1BO/0g8KXhvqiK8puF8JOd+7s5bxdWo9j2UMl6dLDlYpmv +RXFLOwi8VSausvwJDqTI7GfyIM4WDJ6tEpeRokC5Skfcu67gQ3PdizcVnMYh2hKzkF2 fPukdZX0Bprfp0xvnMe1xyYXlxK/AtlQgLCBEeo3M0h8h3pkZ0ugQiWwvn2GyPTfTL6X 9Ei0I4rOK2JTSmRX4llyrlTsf9h5dGQrmU7lnn5hpYvjGe6I9jC2Zl0zAtK9Fy0/vl5V /5zWhE89Dozncfmw/8lo0ItGmXB23FsVVUtMARG6iveQ8lolkMbQrumEtkEk95hYvQBg zeQA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186708" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 09/24] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Date: Thu, 30 Aug 2018 07:38:49 -0700 Message-Id: <20180830143904.3168-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We are going to create _PAGE_DIRTY_SW for non-hardware, memory management purposes. Rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e4ffa565a69f..aab42464f6a1 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -316,7 +316,7 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) @@ -390,7 +390,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -444,7 +444,7 @@ static inline pud_t pud_wrprotect(pud_t pud) static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index b64acb08a62b..0657a22d5216 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -15,7 +15,7 @@ #define _PAGE_BIT_PWT 3 /* page write through */ #define _PAGE_BIT_PCD 4 /* page cache disabled */ #define _PAGE_BIT_ACCESSED 5 /* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6 /* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6 /* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7 /* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7 /* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8 /* Global TLB entry PPro+ */ @@ -45,7 +45,7 @@ #define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT) #define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD) #define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW) #define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE) #define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL) #define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1) @@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3) #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED) #else #define _PAGE_KNL_ERRATUM_MASK 0 #endif @@ -112,9 +112,9 @@ #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) #define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * Set of bits not changed in pte_modify. The pte's @@ -123,7 +123,7 @@ * pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED) #define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL) #define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX) #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) @@ -187,7 +188,7 @@ enum page_cache_mode { #define _PAGE_ENC (_AT(pteval_t, sme_me_mask)) #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC) #define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER) #define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index 11eda21eb697..e7665a4767b3 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 1d26f3c4985b..1cf7d21608be 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -5848,7 +5848,7 @@ static int init_rmode_identity_map(struct kvm *kvm) /* Set up identity-mapping pagetable for EPT in real mode */ for (i = 0; i < PT32_ENT_PER_PAGE; i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r = kvm_write_guest_page(kvm, identity_map_pfn, &tmp, i * sizeof(tmp), sizeof(tmp)); if (r < 0) From patchwork Thu Aug 30 14:38:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581839 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D504F5A4 for ; Thu, 30 Aug 2018 14:44:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C31432B560 for ; Thu, 30 Aug 2018 14:44:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6E892BC25; Thu, 30 Aug 2018 14:44:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB1E72B560 for ; Thu, 30 Aug 2018 14:44:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ACDA6B522E; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7DD046B5221; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E2DE6B522A; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 478F36B5221 for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id x85-v6so4876513pfe.13 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=aL/OXhbuK0dd5n/upzvMhIyYe+n9HeUNkSz6zukRCCQ=; b=C+Adel8Yj+qAkHgSpY2DKH5CMEgWS4ocq1g9CjJ5spi3AESWMzWVmKNKcFPaWv5yKr JpKS8eoTjBpa3zM+8FoulWXBYTWJFTjLsRAF1eq5zXq02fxGxW4vPPw6iPckDot9kp7P tcFItl9ZM7Son8/1ZmUlokW0zCo0hB0YKoIJ7hiP9LDPDvEZb726C1RDTpyqkrS+GbPu 8JxGjJL1u0gHRu1L5c3Xk+uZmP4Mi6OEPrRuCadMCtXzfypSV6LFQUtKLOBKBFe6Tfsk tYtbNoYCCJoOFXKHTnT7NGV/E+J6y9Ek0IPdJaAhRWTcTHFlZ/csp48LOMqqd0o3BI7M 3hew== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AIDCxCw7FkkW7UMRLmUUE68ZmG76reif6Iux8AJj1sRMdI8Kuf qEIF11GKmQc1Hw/NCoRpOXPAjIRYZRIeRp5J/CXpTD4FWYLb21hBMOJlD6O8s0Jaz3YF0By01d/ raNbM/8a8RvOkrvMeBwDvcc7eB47Kodk+xVu4UszO7wqnyKGoAwRkM/4awCR7eb82rg== X-Received: by 2002:a17:902:be08:: with SMTP id r8-v6mr10635832pls.265.1535640225938; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYh7pyXZoLKuxgaD6eaEyGQlDk66+9Py57OYnyTJtMNT/3ysoZWrwm0QDpnSgp0iUiDBxsg X-Received: by 2002:a17:902:be08:: with SMTP id r8-v6mr10635767pls.265.1535640224709; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=Ft7fUQ6d5RaRriKhu6cgon1jolNO38hW0sru0Nt18lBa960Q1S3GFx9YimScgOVMmS oKxAJD+f2BX+/7cELn0ZaUMfyd0qCVdZcMNDZ7fgI0EASsbihJ0kO1AaxuYybVdO1gQ3 KrxB52AjGFT2ILheKHnBoEzt9+39Jmq6i/1lmWOmto13wVmBuwXhbggjUbC7pbeQuMpi wDAeqFTxH2QB/vAu8INoQYboTHtDiBdWGWODaWejm+RlUMELoPh1bwGmgv02VwzQxJ16 q6nVPtC1ZvVBpSmX20ZzYTr2yOUrpXy031Rp9zfLzKIBiYgw+1OaYiCV6NwPD+FZPw9w 8xoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=aL/OXhbuK0dd5n/upzvMhIyYe+n9HeUNkSz6zukRCCQ=; b=KogvvIFrbi7lEz1cl71AQ6noW+I5DReXvgti8Fe5KMUru7jSQ9sZc3JLVhcc56lk3A gSAtXSjsPyuUFohIm+HbspLmAdKu97rt7pOBgc3YsuRLpQL84FPRrS26yQ5mV/WwfSst dbE+8kz0yZ/EzlA2PYrw43GOK6UeG5/Q2ybWj59qXj9wWatTlhiBBHEg7QTdQgVYWc7N ecJCIJRM2bX+b2nYOlqcPShhTIcktj2Z5EOI1dzoGYOwz/rj6JEGTks2oHUVWsR2BLZD Wo/DhqiSzVOKsWwBOJwq66HHoIpTrivbgMav/cB/dygmapXhZ/vvX4WghJR1QQ6i3Owa ZiBQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186709" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 10/24] x86/mm: Introduce _PAGE_DIRTY_SW Date: Thu, 30 Aug 2018 07:38:50 -0700 Message-Id: <20180830143904.3168-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A RO and dirty PTE exists in the following cases: (a) A page is modified and then shared with a fork()'ed child; (b) A R/O page that has been COW'ed; (c) A SHSTK page. The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings: Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW) Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs. When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 14 ++- include/asm-generic/pgtable.h | 21 +++++ 3 files changed, 142 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index aab42464f6a1..4d50de77ea96 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -119,9 +119,9 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; } @@ -143,9 +143,9 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } static inline int pmd_young(pmd_t pmd) @@ -153,9 +153,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -294,9 +294,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + if (pte_flags(pte) & from) + pte = pte_set_flags(pte_clear_flags(pte, from), to); + return pte; +} +#else +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + return pte; +} +#endif + static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -306,6 +320,7 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pte_clear_flags(pte, _PAGE_RW); } @@ -316,9 +331,24 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { + pteval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHSTK_USER) || + pte_write(pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + pte = pte_clear_flags(pte, _PAGE_DIRTY_SW); return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw(pte_t pte) +{ + return pte_flags(pte) & _PAGE_DIRTY_HW; +} +#endif + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -326,6 +356,7 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pte_set_flags(pte, _PAGE_RW); } @@ -373,6 +404,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + if (pmd_flags(pmd) & from) + pmd = pmd_set_flags(pmd_clear_flags(pmd, from), to); + return pmd; +} +#else +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + return pmd; +} +#endif + static inline pmd_t pmd_mkold(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_ACCESSED); @@ -380,19 +425,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pmd_clear_flags(pmd, _PAGE_RW); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { + pmdval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHSTK_USER) || + (pmd_flags(pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY_SW); return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_DIRTY_HW; +} +#endif + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -410,6 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pmd_set_flags(pmd, _PAGE_RW); } @@ -427,6 +490,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + if (pud_flags(pud) & from) + pud = pud_set_flags(pud_clear_flags(pud, from), to); + return pud; +} +#else +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + return pud; +} +#endif + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); @@ -434,17 +511,22 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pud_clear_flags(pud, _PAGE_RW); } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); + pudval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHSTK_USER) || + (pud_flags(pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -464,6 +546,7 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pud_set_flags(pud, _PAGE_RW); } @@ -595,19 +678,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); + if ((pte_write(pte) && !(pgprot_val(newprot) & _PAGE_RW))) + return pte_move_flags(__pte(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); return __pte(val); } -static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) -{ - pmdval_t val = pmd_val(pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; - val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); -} - /* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modify static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) @@ -1159,6 +1235,19 @@ static inline int pmd_write(pmd_t pmd) return pmd_flags(pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + pmdval_t val = pmd_val(pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); + if ((pmd_write(pmd) && !(pgprot_val(newprot) & _PAGE_RW))) + return pmd_move_flags(__pmd(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); + return __pmd(val); +} + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 0657a22d5216..f47bbc1f9c45 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,7 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57 /* available for programmer */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -34,6 +35,7 @@ #define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ @@ -109,6 +111,14 @@ #define _PAGE_DEVMAP (_AT(pteval_t, 0)) #endif +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ @@ -122,9 +132,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) /* diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 88ebc6102c7c..aa5271717126 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1127,4 +1127,25 @@ static inline bool arch_has_pfn_modify_check(void) #endif #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + return pte; +} +static inline bool pte_dirty_hw(pte_t pte) +{ + return false; +} + +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + return pmd; +} + +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return false; +} +#endif + #endif /* _ASM_GENERIC_PGTABLE_H */ From patchwork Thu Aug 30 14:38:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581841 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1028D14E1 for ; Thu, 30 Aug 2018 14:44:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F36AC2B560 for ; Thu, 30 Aug 2018 14:44:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E74D92BC25; Thu, 30 Aug 2018 14:44:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 794932B560 for ; Thu, 30 Aug 2018 14:44:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF39F6B5223; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BED596B521F; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 471D36B521F; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 785606B5222 for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id g12-v6so4067923plo.1 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=NHXpEJZmh0RaPkbDE+qgSdLq1VPEfGibunzGbqN5OLEcUt7ug4j0R/Kbck4wsVBgE2 bIdHJ5cz6VagsfcbbtQEHY1uPLMHwHLcMJ9D3LjdFv5lS7Fyi+mqlFKcOu7jVWTkos5R JcIeiO0ZUxHExnfahz6TTN6E6E+q1GCQ0UxiOeAI+rpw0pSg6nlxxy/xxmRBLs2kMtOA gDNdSoxuBUNoj7XJRlHG3HoOVSWbcw8jvsAetS4XfQgZ3wCutsXqaSrqHLb7ZWOgj5aE bRFIYrMxo2Rk+EABb7I6E+SG2dM2ee8L60IH0PEEhOXq7End95AQ82ujCXiyOvGDRmqB nwXw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AUS6Ah9FaVRwEWkfX14go/zlNn5V/NEBT6fxRIrTnKIyg6RUgS nu9so6LXwX7aYKk5+bV9K7Id1BqXHmiHGSaQ1ctVk+t6L9EKpK4D8V6l607o35X/n4/rfk5Whz+ roMxdaZH0LN0Br3GUmvNWM/VOKzCmW36qV/e/YbLoICegXewKQ1RhfnltwdkrVQnCxw== X-Received: by 2002:a63:2c0e:: with SMTP id s14-v6mr8334561pgs.199.1535640226149; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYwDdLJI8toeT65h4ujfSh0vJTz5Cp1VqYpGhuVteHhBmgPnJPO+U7yXzdxztvQOMveFn3z X-Received: by 2002:a63:2c0e:: with SMTP id s14-v6mr8334521pgs.199.1535640225236; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=fj7QEwVzs5Mr6TnSoDWfZm60ZfA/16cPoW/diJN67KKsA4g8UMMo1eNScYrNsInYp1 BEgtLIxYWAz9EiNEZOSQOPjqH+mMwIJxi9r0XXjNnIVrK9soMZY2hoRx/jqSVbgTsbWm W4bqn+o1PazPmTdBw8BNRp+dUAqqrmPFIOOS48m9P+dpQMDmR0EwlhdcqawArykQY5uD MCzRr1P23zc5I3b+xZcwRFfWgtFiTXeQis9PtEHLn0BWH5aazkObYolBWW6OZ35USfxn 8+wsXj1/JJEVXcs9LicN/EpgsApbddNnxwnb0oIbIAC0fb5Jed3aN2Z4VgU6MljaFKEi 60sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=jgmagsFc3rP50Xr0rnIZD9owkCV9gX69fCyrt6L97r7agBSY6dyDI0qesWJJuhlzQt /IF52OKhW4CrEIXI4TzpdAh2Q4LqqpRpC+09YH9Zm5FZ437ZstZsdN6kcw1O2/VgH5oK 85dEHG98uQFeOcozZwsMGuAGG5t1FcVKlgEgnoxpkwLouP3KkQrs+KVIULnNcCpwG+o0 ygz0hQx7K/G3w3W0Js8uca9eQVa0rvV7ggG4OaAaZBKvzin6cyX18MGv3ARVnLlV/2ec M6DGr3b9Gm0/iM8TindiD3xoNmhAbi8QXU83JtkxOuM/MXU4Mr/wgzVnS2aKm2XZTNSC TufQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186712" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 11/24] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Date: Thu, 30 Aug 2018 07:38:51 -0700 Message-Id: <20180830143904.3168-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Update _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry(). In order to support Control Flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW. Signed-off-by: Yu-cheng Yu --- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 00aad8164dec..2d6ba1462dd8 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1170,7 +1170,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, } /* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse(se); ops->clear_ips(se); From patchwork Thu Aug 30 14:38:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581845 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BE645A4 for ; Thu, 30 Aug 2018 14:44:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE4A52B560 for ; Thu, 30 Aug 2018 14:44:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E1A682BC25; Thu, 30 Aug 2018 14:44:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74B092B560 for ; Thu, 30 Aug 2018 14:44:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 684416B5226; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 47BCE6B522B; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64A006B5228; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 94DFB6B5226 for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id d40-v6so4052464pla.14 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=EQNscwzpy+cjA0WX6EmQ4Rhmb4cUPgV+Kog5rBSI4mM=; b=SVhmLnUziHjZfdGwWmIwXYX8XEuHQtNYutV5MlqJ4SL9TSeRJxWl4PVYL5i8HlIq+J V7r+2vaFCFP1yz3NxbWwHn58Q5ojAn1jvhVaS/cYipk7DMkrbTPjYqDk6FQG19JAKPlx +vtBwofd02ZfYA64fHGXSpEp5FZoc44rwOMQGb9QsmpbHwgjikRbct6RdUl6yBxy0yw1 71NNIz11xpAJqJA/J8Wm3nDkf2j1Rl0J6R2Y1L8dhsh0cE7ZxjTsYw+J/EgJoG4d6KP8 ldBVabAqLa1Tt/zaAtP1HrKN+FRa4vw7MYCkXjC7rhHGpFZ1PjrhCLkyffMb2l/d4dm6 v75g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BGU0wcn3UMVXDioyPfsLp3PdNtmtKGn4Tufrf0uVkv+Btbieh7 eyrnlPvNgVI7hON1uH8S8G7rDwpGJD9sms0NsPGO38e+DJ6WYKQOlAiCfLq/PARCdQJQuNx2L6T o2Ckdt3UNSwlJzoMGO1ajPSrZA8CHOrC8NMSkmbo+FdjcE2r1SQyClJAD7VCxYdx86Q== X-Received: by 2002:a65:594b:: with SMTP id g11-v6mr10024010pgu.260.1535640226269; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYgmXVsuxpu4GugfSGyzll4bdqetTBcpUMvJacRjxj387f7L2wAhTvC3g1ztzeJ4fgK09DA X-Received: by 2002:a65:594b:: with SMTP id g11-v6mr10023951pgu.260.1535640224955; Thu, 30 Aug 2018 07:43:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640224; cv=none; d=google.com; s=arc-20160816; b=gpO70HF+uZEI6YJofNzlDLM3t60F1Clqxi9w8jSD3hEGdZqU/lvvo9XvINpLpQ9Zyj pjl8fZz5XiktWcC4y8+c589Y6TAXTcZKwJy15/6qgoZG9aVUz41pgwk+Mb75dR73wbdU McvUAt3sC65AzF5qPv2e78f7eTR1CMU8FIiBcOQVxnS82Vxp+R3xDwqJ+NBGP+/4ulrD f/pbX+bWrAcA0wchKX7vJKEjus03V22fBQv0tAbt3WAqwFHl4lVFvCMzxs2ShjScd6qW wVrrG0b9TE81tQozzbvxdiMrUNf4WOC53pkIyOv9JWxMAtSm/ebBHby8+2nkThCw3YRD 4vkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=EQNscwzpy+cjA0WX6EmQ4Rhmb4cUPgV+Kog5rBSI4mM=; b=CmqAPC+UDVsQ1BbwNvCnuZoU5tI8t5qQ13JjWMDtFMJEyWuTR4LkJwYKNiek0886Hp anFc2lTe17QPMuOavt9fsIn2RBKQaYwb2Utbx4LYGhv59RPZJH8sumV4Mati00AXREYC iL5Vz9Vmouhj5K4tFgFJEzr3cX6of8zeBYsI9PrRq7G2iPUz1IpHE7YmFwq4Kw2IUMAR C8cipUmmgVx8gq3u6tlSQarUHqyZH4OMt5B/hrqY49nspAyheZYvTRU29Bav3+zvA7RE 1b895B7OM3MAapUqLndLuIMrwvQknohQrsF/Ue2Znj5qz9S6Vdj2JumqNwYg8w9aDrHJ SqEg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186713" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Date: Thu, 30 Aug 2018 07:38:52 -0700 Message-Id: <20180830143904.3168-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When Shadow Stack is enabled, the read-only and PAGE_DIRTY_HW PTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we use PAGE_DIRTY_SW. Update ptep_set_wrprotect() and pmdp_set_wrprotect(). Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 4d50de77ea96..556ef258eeff 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { + pte_t pte; + clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + + /* + * Some processors can start a write, but ending up seeing + * a read-only PTE by the time they get to the Dirty bit. + * In this case, they will set the Dirty bit, leaving a + * read-only, Dirty PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and will not occur + * on processors supporting Shadow Stacks. Without this + * guarantee, a transition to a non-present PTE and flush the + * TLB would be needed. + * + * When change a writable PTE to read-only and if the PTE has + * _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so + * that the PTE is not a valid Shadow Stack PTE. + */ + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); + set_pte_at(mm, addr, ptep, pte); } #define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) @@ -1266,7 +1287,28 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { + pmd_t pmd; + clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); + pmd = *pmdp; + + /* + * Some processors can start a write, but ending up seeing + * a read-only PTE by the time they get to the Dirty bit. + * In this case, they will set the Dirty bit, leaving a + * read-only, Dirty PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and will not occur + * on processors supporting Shadow Stacks. Without this + * guarantee, a transition to a non-present PTE and flush the + * TLB would be needed. + * + * When change a writable PTE to read-only and if the PTE has + * _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so + * that the PTE is not a valid Shadow Stack PTE. + */ + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); + set_pmd_at(mm, addr, pmdp, pmd); } #define pud_write pud_write From patchwork Thu Aug 30 14:38:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581835 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AAD814E1 for ; Thu, 30 Aug 2018 14:44:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE0ED2B560 for ; Thu, 30 Aug 2018 14:44:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E0CA52BC25; Thu, 30 Aug 2018 14:44:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76C892B560 for ; Thu, 30 Aug 2018 14:44:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11EB66B5222; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EEF9D6B5220; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF4DB6B5220; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 3771C6B521E for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id c8-v6so4886477pfn.2 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=tUVERZhykosIqoOxzOVgpuEQzQHuZq0XBD5oE1YaqTA=; b=GVywtPD+vhjUonBkwloUhwWn1pSBseYrSmxb2sUc/CMJbq2D0TlsScexIN+z7QUnXA 0BOuHPDthYJNJds4LbxgI7g+c9W56U2VVCjgHeP5ymAslzI5yKhIFHphua6C0oYF2WaB srFARDGxVcBqn+uquko1TCJyVKBBXRWw+wYrIIhZX5tK7xjJlw52LhAI1/Ds0xbPcYTT OJ5yVmMfBrJ6RSfOOxDov7HnuGcxK/jPB+if3eyDz7iOGUhELwpeorWK3Kg6CKmYTVbm 9nNdVBDu3rig8jdWdhARyJdvN+rctctkzlKMb6t0FprSjlyJYfRpRLEi+uxtuuD++Jx+ Wjqg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DfxrLlAVrgWTrEn4yKICTaB5XDpo0fj9MAmgWBSAlhuxj9aa8B Lm24MZ94dR7Tdd0iDbnFfrI4dyCEVC0vaE0KhOdRegT6JE3y5LXkQF9nw8VILRROsiTMLT+JW0k 9yT5Udcl0KfRRU2OY/TNLTkf7jWllISABDPTpguzuHw5nO/7SWYZVZx8RohFO0jbUNQ== X-Received: by 2002:a17:902:c6b:: with SMTP id 98-v6mr10601486pls.233.1535640225896; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYflbBCpU01DB2wy9+Q+RRB8Lq4qVfU95pZ//BeeqIRfmN14dvPDgrjm3/SX7gqvkX2oQse X-Received: by 2002:a17:902:c6b:: with SMTP id 98-v6mr10601454pls.233.1535640225158; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=GQR26qvjmT6FV17gxq1vBmshOVJaialb8yE3S8mAzlItJYU54Q9Vv1JaNnYMbbw4mW 5tq2zM+vzH0cvN8vqNWTLyi+KiJ+91fyHmebhzURlulRiNJMudRnObXAAbp3goRZh0EE opzNxfVmKIt6M+nV2AhRp5WDvNu6zY1OTGfY3yQcVzJ2uQWv1FVHWAYKzI4fdvNE75Qz +TqL3VqV9K6eVCWWyfCmMrTEoDlTbB4fWGiGdHxUTSrprIBQxRvmiNCPxu5SWUB7z7LF 9jmsKg4dVZxNjydcA5HuKsSLXO//wcObULrKWa6iDqANhh830lbByN4EUTw0fgDPJe5I i3dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=tUVERZhykosIqoOxzOVgpuEQzQHuZq0XBD5oE1YaqTA=; b=vph3LquthaaxhYQsWleZ5YkRu2z6d3pJFcJy+hq3H957b0pBeWw6GFI0N8jUNb85Vo HmMUn9Zy+hhOWMrshErIcOMCUR50TKbfuVKjTmE+1UcVtslo27/UpnKcyT6C4QxCzcp7 6c9QUPRQS4UAYeKv7RGK9/9PSNCHHysxOA6Nj7gtKz9xNmruaEKZQwY3yeuUYypO4oQ5 wNXoCuPHmyn7W92ukeWSfNNnKauzJLZR9l+jeUCXQjK1QZ0nr0zQBCgE4r9Kmy/KR0Ss yICQ0BT+yeNsw/byE6aQ/pu6MDa01/xi1o39cWHFkQNi86SLRusdsn/1Uo0nPiCuA+b4 qrQQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id t2-v6si6809115pge.64.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186714" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 13/24] x86/mm: Shadow stack page fault error checking Date: Thu, 30 Aug 2018 07:38:53 -0700 Message-Id: <20180830143904.3168-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set. In access_error(), we check if a shadow stack page fault is within a shadow stack memory area. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 5196050ff3d5..58ea2f5722e9 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -157,6 +157,7 @@ enum { * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code { X86_PF_PROT = 1 << 0, @@ -165,5 +166,6 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, }; #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index b9123c497e0a..3842353fb4a3 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1162,6 +1162,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Verify X86_PF_SHSTK is within a shadow stack VMA. + * It is always an error if there is a shadow stack + * fault outside a shadow stack VMA. + */ + if (error_code & X86_PF_SHSTK) { + if (!(vma->vm_flags & VM_SHSTK)) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ if (unlikely(!(vma->vm_flags & VM_WRITE))) @@ -1300,6 +1311,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * If the fault is caused by a shadow stack access, + * i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set + * FAULT_FLAG_WRITE to effect copy-on-write. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR) From patchwork Thu Aug 30 14:38:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BBA60174A for ; Thu, 30 Aug 2018 14:45:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAB902BC83 for ; Thu, 30 Aug 2018 14:45:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E44C2BEFF; Thu, 30 Aug 2018 14:45:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 298D42BC83 for ; Thu, 30 Aug 2018 14:45:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB57F6B5228; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6E9526B5231; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7AF16B5228; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 68E3F6B522D for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id g9-v6so5141607pgc.16 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Bzwo99A4d1pJDwCeY9DYj8i9R2GNeR2Ec6iJjwicYHg=; b=jo8K1LInyHhFTozDt6ZlYWwhrxlttVYQ0PxcvVoX6uEZQJS5TZS0szd3AuZr5ALGMY rkioWB483TK7vK2JKOsAWi1zESHaFStwdZdRJjpGowGkLLeg0VVacRkf3Q5PfwQeDdfc TaiXkRzCG3S0re79ByDOyrh9LF9835Xv0DLTDDJsVuqLO2gQbp0Hk+g0Jd0Jerse9YOd 7anD1pb1Adh6Amw/ctI4zmtWp+o7Y9djQ+/acMtHd2PX71iA0IqoxdiY0Hv74heWu+qG b105FuwzWfwDVKhpsTrpwFlMO4LjxGjzXxClL9kdd9CBge38FjBikhvASc55hy027eFC gBcQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51B3PZ95O4okIlGJ96u1UzwmEAROcq25YKIPl/paiTsUHPLw3Y1W dA07sI2FbtY1loteqmXe0elEeHorU5Sk+yabigJ6WTPpBdoGCt6pjOWzVPhQ3jw8GQBw32wwGfN L81SkWekVyobr3pxW8nL6uHxI6wCaA6fXZKC1bQULXbFRmvjOyj7HTqRzakT2DDuxDQ== X-Received: by 2002:a63:41c4:: with SMTP id o187-v6mr2409935pga.297.1535640227111; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaVUWvy2q9YlQHetkG9Ierznr/LkNbUo6Ik/doRAGyyiWhJp6yKCL/de9ShDXvHxfJYsBFK X-Received: by 2002:a63:41c4:: with SMTP id o187-v6mr2409902pga.297.1535640226390; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=I8+onYElEGB2Cnbvk404xXjdS/oc5OJtymuZCIG/WMobbd7s/FbLeCdeTD0gmlz9Ag cMTIdY5ilI80aYOn8+3bSHykEolvutygH3ByBG92TUv9gu/MayngRxOTQJO/l9jLlh1N 74ihdTCcUJjlS4kluTnVieKBFVPL/nkF3z9OJoRdwbq/je+Uj9ufDEkjHqiZrtTRq3Kk xn7vyJmkX+uh2p4EX8DbHrBRnE+YJ+xowl2P0a9DpdE+UA2EV7yoyy4Om8MRT2S1/lev WJOx4Bn/ZypDKm1TPktXycXY1Gjo0wEtFwz2ZDbH+Zei/1J9LOZv1OH3pUMRJumwqUBo qrbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=Bzwo99A4d1pJDwCeY9DYj8i9R2GNeR2Ec6iJjwicYHg=; b=YBvKyxJo4U7Q4xxzkgCeGxwRF1AnUb3HyAGIhwehy5FXa5hBdEfef/ZixDaqKV8feM +M9dicJ/uBi3QBpXqLRHKjD1kscSxeb3N5OMZvlRJTdzQ8xN0DqwrFtchlScknb1YyAA 1bPQyGgzEUKWNNFioaFqJG3xsIjhztCV+eycaB0r85VYDCYpmdxEe/4JA9i7+yd6M7I5 pq8vSAQnJh9975FuUjAGqNBKbVXAR9ZCZwuF9isBwVunHUnlRG/DkZXqjv/sDSm2sopg a9+aq2iFLcbVFI8bKVMkQnfur6mAtv8hXUkrE11xhKDSS65p4lID6RtXUzLnaLRoKsQj Ua6g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id z73-v6si6610727pgd.471.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186715" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 14/24] mm: Handle shadow stack page fault Date: Thu, 30 Aug 2018 07:38:54 -0700 Message-Id: <20180830143904.3168-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a task does fork(), its shadow stack must be duplicated for the child. However, the child may not actually use all pages of of the copied shadow stack. This patch implements a flow that is similar to copy-on-write of an anonymous page, but for shadow stack memory. A shadow stack PTE needs to be RO and dirty. We use this dirty bit requirement to effect the copying of shadow stack pages. In copy_one_pte(), we clear the dirty bit from the shadow stack PTE. On the next shadow stack access to the PTE, a page fault occurs. At that time, we then copy/re-use the page and fix the PTE. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 10 ++++++++++ include/asm-generic/pgtable.h | 7 +++++++ mm/memory.c | 3 +++ 3 files changed, 20 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index e848a4811785..c63261128ac3 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -872,3 +872,13 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_mkdirty_shstk(pte); + else + return pte; +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index aa5271717126..558a485617cd 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1146,6 +1146,13 @@ static inline bool pmd_dirty_hw(pmd_t pmd) { return false; } + +static inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + return pte; +} +#else +pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..9b4e11944b5d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2462,6 +2462,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2535,6 +2536,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -3187,6 +3189,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_set_vma_features(entry, vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); From patchwork Thu Aug 30 14:38:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4240514E1 for ; Thu, 30 Aug 2018 14:44:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 30BE62B560 for ; Thu, 30 Aug 2018 14:44:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23CCC2BC83; Thu, 30 Aug 2018 14:44:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97CA12B560 for ; Thu, 30 Aug 2018 14:44:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2839B6B5225; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 19AA86B5233; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E87B6B5230; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 527A96B5225 for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id j15-v6so4833822pff.12 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=tU4mYnDOs/8FLat8c5d4nBt0YTG3mI5RrXgj3u0mNnc=; b=oTmE4LGgdLpZJQsuGZAcDUfOMGf0C34XHoR0WagQ3pITwKNAIKhoL7vR307bbTBG6a AfgMluMUaC96LCL7OY4xcK9fXXIgg8aEa49JDESSFMjRervOJsDsLZ8iuPAUdiSdlIvl pfRf/ZLtlmsK4ebuQ7GU+BHtEJeD1tnsw2n6pwYr2ENAdxZ/uT27WPFWgmvGSw6wcnUH +pyWhOr34lHCXOGn9ACKrX405eKkI10LAAWSDt1PKgQuH8ZywlF8oVyYnpoPM3uISvh3 h9ce2QRiAGdQgv0GVTlsZ3HEBF3KQAk8b4VPzBkosX85RbOkaEdluTPjVktAgy8H2jAH 6zmA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BL9uKkki1SwkrfL9S5oK4DeqM8KpoGqqaIy0H5wt9PQGBqQevQ TPmnDhwqV7yKjnz7ukdckd2ItokYAkLerIpAWth55TWHrMATtPU1oOJSsD6PgsunP+aUeJGvl6f pEG1D1tsp1n2xD5/KY9W6kvWmnPhJZXl6BD3i3mVjSj33I5+SRd4o/HczHEYjy9IRrA== X-Received: by 2002:a63:2c0e:: with SMTP id s14-v6mr8334602pgs.199.1535640226999; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYRojN7vI+kPN+MCCOg7n2L5A8HAX1mh/0n8Qhg6PmkQsPNeGkSC8exLpNsUE/vS/clCnZx X-Received: by 2002:a63:2c0e:: with SMTP id s14-v6mr8334539pgs.199.1535640225739; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=KedmH6IKxcbEpqSTWUXms3bjqlbSAIS8vLBdxDh3Qvs/NRavM1J+hUHu5XafgB6T21 hvVV0EuylbcGP6MmjzpfzifJO381WT/a38IgbXH+rovrxVsivJMWY3Jj9W79Ch0a9Ylf C3CEjo7+9Bqa5vamNchqpGwJahMaWdkcezRGHVSdsOrePXqjq9Q/OPgT9VOlMdLMTgSt WC/Op8lZKfZi0TLWmKFF10v/gYuiV0svnlOwxm3cOzqK6ykYjTf17VM6b0p7li+bamNA tHkufUFI/Dj3iDXMwv90PIJ4C/du6DXEeyc5PB2FWklVE3fF7BayYiv2jyxZsLU/NovN MPtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=tU4mYnDOs/8FLat8c5d4nBt0YTG3mI5RrXgj3u0mNnc=; b=dc8eXpIdZ3qfkbrXza5iRIoKbyu8y+9sM5W/jncMpBQhs3OF3+4n5wPhdYrWWy23qv V+YoKK7tyuChbcEfBjb8Fz1VSQJ1uFOBes50lLSSqvrCQ1o5mfNSdM1uGW8yLvkyy5kX DNUti7JRnVYEl1lx1vkGVXE7C42ud2svIKdJZEFRgBKnHz2ni9+jZNCIdlRyIFNUxHmu gNoOqX9taYvZH3xMbpkW5wOx1LKmmKz8F7/IivKKOb+Bf4qDf/YvgKtr0Yz1JvIySvNo Qv9+GbLXgs3ZpjVDuUIMKWMnZeuYnhhN7ItyAva0QIC6VEY+80Lq0xuADvwgbFoQEzI4 ek6w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id z73-v6si6610727pgd.471.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186718" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 15/24] mm: Handle THP/HugeTLB shadow stack page fault Date: Thu, 30 Aug 2018 07:38:55 -0700 Message-Id: <20180830143904.3168-16-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch implements THP shadow stack memory copying in the same way as the previous patch for regular PTE. In copy_huge_pmd(), we clear the dirty bit from the PMD. On the next shadow stack access to the PMD, a page fault occurs. At that time, the page is copied/re-used and the PMD is fixed. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 6 ++++++ mm/huge_memory.c | 4 ++++ 3 files changed, 18 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index c63261128ac3..0ab38bfbedfc 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -881,4 +881,12 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) else return pte; } + +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 558a485617cd..0f25186cd38d 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1151,8 +1151,14 @@ static inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) { return pte; } + +static inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + return pmd; +} #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c3bc7e9c9a2a..5b4c8f2fb85e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -597,6 +597,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1194,6 +1195,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1278,6 +1280,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1349,6 +1352,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); From patchwork Thu Aug 30 14:38:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B59EB5A4 for ; Thu, 30 Aug 2018 14:45:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A60722B560 for ; Thu, 30 Aug 2018 14:45:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98A142BC83; Thu, 30 Aug 2018 14:45:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09C2B2B560 for ; Thu, 30 Aug 2018 14:45:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0A036B5220; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9F8596B5233; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8A536B5234; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id F070B6B5231 for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id r2-v6so5168109pgp.3 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=DS339Oq/VMp5xvhucSCa2NrKJaq5mpRNUbsXZO7WFgw=; b=AQLep8Rw838RR+Umax8lbl2PJzgz4+v32NwwOn8WNq0rC3Vpch1NXUgPPrspcSaEV8 eskR2JOmZS53TFRS8RYe4ENAMoIl7s0u2ZFqT/P7TYe8XqcJl05nJsGNNAGqeiQFH9H4 QkLFkhBwwQsSgRLKG3mir48UTdtIAxtb9GIm/M7wPGH8mWeVqYJaa++ZlK8ZGzHzJ9wU J9bCABu4LOiX9Pbi5QbXgIPOumASpw8+ktfzF5w3zo5n+DeNvMOHu7PepgSMG47z5ZiX HOfevDn+2KnMsHHKFoiSuTyENOmeSYJNEbUfRPaZW+EO9HDW6u9gZZL5Ufn2Ftx9lixq Qn1A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Duyo0Oy1M66z8/0ivwVjvxEVQaYHNEpRoNAZc7yT7q5HQj/Ay3 SaqiCQmM3pUVgOAD5l+bUZ6BTR/VL/ou2tQU6D/DUJfp6dJ+ED85q+lfA9xcJjzk/qzGQeUyxRz mCDaggFLa6rU/zG2WJGhaBQxtbFdaK8aADYMHsk2AESHhsb+onUoCeX8JhmkjTn99Gw== X-Received: by 2002:aa7:8087:: with SMTP id v7-v6mr10757683pff.38.1535640227665; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda+xVA/HMjZ9PUIefWP0RY7Ky+GP/51mvN8fhilbptk0YIZk7dU71vr4bLFaECkTeCZRyo1 X-Received: by 2002:aa7:8087:: with SMTP id v7-v6mr10757611pff.38.1535640226387; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=J/hCszZEiHVB5q4yDjADsR8XWtAee63jIk8F8CDiBBVRfOid+1AVAbOwkCQYzWnLNM pftD4y9MuGHqqWp3ASRqcKTgqCx+/WID9Tkq46MDvE1vC+tqd+KX6NGDSW1BmzBrQoEn WScwPrTQBBVnxdk77DV+mq0G/ltLbItCo2EphmKZ1QznmKNO/q5B5RhkWJqxDDTGBBG6 eBQ7nF830tR3oEuPOk/X1guhuZqy7kOX/XPnHM0Onn4uBbwdD5aIUD4YoVvtAbCmnvKC 4YA0yn37JEQcLc3d0pJYy2tQtHMTpopzeuI/Tu5ZeCK+hfwH+uif6XLqJ8u1Fiica72E cy/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=DS339Oq/VMp5xvhucSCa2NrKJaq5mpRNUbsXZO7WFgw=; b=oMU/BGi2wl7NbZ8rfuYaChQoQfpgOZrTaCNP0y0x5QwiPp7m2z99H9NCHzWfMfpRj1 riqjg5gXT6fIq5FVb6C+dZpYjK4tEMCf/LNyZLOxIEi7SHB3l/SUxcaBVGzh0AnlANsO LfoLcJanQddB7+NMkRvy0qda2/APXjYl6Pq4TZZ6zcpcaoJ/wGHuBd1ropUMP3cYWOS5 ndzFQrA1XMiJ6RmpJnH04VC/4rTOsWr8glzF04IwSh/5FUzy2617NGr5b9BnzuQAkirt KLYDBiqWSd1y08nC5Kg6PlyqsUoPpsw8cebJMXLs5zo3pbDDhLH/8XHttu69GS+0dWHU UnCg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186720" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 16/24] mm: Update can_follow_write_pte/pmd for shadow stack Date: Thu, 30 Aug 2018 07:38:56 -0700 Message-Id: <20180830143904.3168-17-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP can_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW. A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW. Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive. Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?" Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 18 ++++++++++++++++++ include/asm-generic/pgtable.h | 18 ++++++++++++++++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 46 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 0ab38bfbedfc..13dd18ad6fd8 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -889,4 +889,22 @@ inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) else return pmd; } + +inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_dirty_hw(pte); + else + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_dirty_hw(pmd); + else + return pmd_dirty(pmd); +} +#endif #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 0f25186cd38d..2e8e7fa4ab71 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1156,9 +1156,27 @@ static inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) { return pmd; } + +#ifdef CONFIG_MMU +static inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + return pmd_dirty(pmd); +} +#endif +#endif /* CONFIG_MMU */ #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); +bool pte_exclusive(pte_t pte, struct vm_area_struct *vma); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma); +#endif #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/gup.c b/mm/gup.c index 1abc8b4afff6..03cb2e331f80 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -64,10 +64,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte(pte_t pte, unsigned int flags) +static inline bool can_follow_write(pte_t pte, unsigned int flags, + struct vm_area_struct *vma) { return pte_write(pte) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pte_exclusive(pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma, @@ -105,7 +107,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; - if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) { + if ((flags & FOLL_WRITE) && !can_follow_write(pte, flags, vma)) { pte_unmap_unlock(ptep, ptl); return NULL; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 5b4c8f2fb85e..702650eec0b2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1387,10 +1387,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) * FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags) +static inline bool can_follow_write(pmd_t pmd, unsigned int flags, + struct vm_area_struct *vma) { return pmd_write(pmd) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pmd_exclusive(pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, @@ -1403,7 +1405,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, assert_spin_locked(pmd_lockptr(mm, pmd)); - if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags)) + if (flags & FOLL_WRITE && !can_follow_write(*pmd, flags, vma)) goto out; /* Avoid dumping huge zero page */ From patchwork Thu Aug 30 14:38:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581843 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 365E35A4 for ; Thu, 30 Aug 2018 14:44:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 259F12B560 for ; Thu, 30 Aug 2018 14:44:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 194662BC25; Thu, 30 Aug 2018 14:44:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF99F2B735 for ; Thu, 30 Aug 2018 14:44:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 226576B521F; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BA1306B522F; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50E916B5220; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 885BB6B5224 for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id i68-v6so4881669pfb.9 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=nuX3QM8HX4YtIzApwaJeEjWpSb/A/OxmLnTA/JbU0aQ=; b=Aqsomi6nTlcGjO7LXIMtfUiHC3m5VhayN5A/IXwU4Btr9hVPVL5kcSdUssicICcjQQ H4/nDTwUWyWEzIC8j+GnYFApVwgIoIo2SeMylGYd1E1OQZ6G9WYfu11o6XhGA1aMacI9 1k7rreMQZUmIqYdLFVwbh0xvR7p7E+NMnZdCtNaMTH8zsi9eMcfMlrXXQWDdo9tZiNNX f+dao2sVmNO3aUmFXmQXEjzqgVaTENirOOcp6a7CCNVy66Z7kcyfvnl+4tyNV+25yv9x ajoScv/x6obrQ8CE+9+5ohJ/a5fkSoDRQDHH5hWYz40f2wtXbinCFGufG1xEylCyN4f/ rHEA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51D4AbDAHNp+CseUHQklbO8JRhEL/HXRSE8w5AamvyfqJNAyCUEe oSdBQGzlu9TwgIdFsdJ6Qu13fdcXFDSY6LaHfxRiUz7UQnf/L9iFya2Tk+u5ge7rZqhUqRrbv9X QlwUNEF0h0VPXYonTqPerkdSouPhYGVN+E3y5ZAGuEmM8Uv4llsRw8NA6aYLh43J4Zw== X-Received: by 2002:a63:9551:: with SMTP id t17-v6mr4272955pgn.432.1535640226241; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaSDby0Of7I2VOfSoi+rr1MZs4ixYI6vCRNxCC5kvfNTicTIOVQO2OkiCUSAZ0NAP8SpsyO X-Received: by 2002:a63:9551:: with SMTP id t17-v6mr4272927pgn.432.1535640225666; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=DO4USj0HJvu+jkV7k1umNoJCAxswHIoZRe6oKvm1oDHywVBnB6oVJUo7ZOX6Vk2lB5 Ry69xrJmwkG3to636Zwrm8GPfQibqWmxST3KE3IlNXYTOdsNvGoSf+ZRETHStM3CXVpM 9ZtEycilkLCvc4ZgiUxUOUZdy6jAXwkS+tcpc/Ya0c/uBNb3uxOs4zyoYff/PDyQists eTzDIpQANA11H0mnflb9uX3Ska7SEqdEi3QN/Z8XvHeKxT1TK96nZhP2jVQ8/vIVaWYQ gRegpFV1tACNQnJ+dnqqGimgiwZosbAK3tLDwbUAIj2aZcUQj/9yyYssa8udZELWLDZI tJ6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=nuX3QM8HX4YtIzApwaJeEjWpSb/A/OxmLnTA/JbU0aQ=; b=LuW2konFcgxM35154FHfAWFpELOiiCpmwT2XptOYmmDW3DO3qXRb+MPwz2OfWfovpf PH8eyl5I/Fh0eOPyp1Vt5FRMwCUnrwCiESpdKM9mFQ3IIoNGRi4fqW/ow1tWHk4TiM9h 7zbk9Dsrd2EMSVfNy2FyxkXlPR7hCXsOw/NaCcWOdEbFBsYXuOXUKbWCFEe/NeOX80m/ hBrRt1sAPnJeHdMAQ8TGmETzatwu/z5OA4Yb0VZwIKd359URUEoDl3yJBnzas55t/RvS IF6r4LAyB1b1QFpEy46FV5NrdKOayQfjoVLw73gwiJeHD9a5KN2rFN6BFokeeeo3W9xh Tg0A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186721" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 17/24] mm: Introduce do_mmap_locked() Date: Thu, 30 Aug 2018 07:38:57 -0700 Message-Id: <20180830143904.3168-18-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f40387ecd920..c4cc07baccda 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2318,6 +2318,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len) static inline void mm_populate(unsigned long addr, unsigned long len) {} #endif +static inline unsigned long do_mmap_locked(unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags) +{ + struct mm_struct *mm = current->mm; + unsigned long populate; + + down_write(&mm->mmap_sem); + addr = do_mmap(NULL, addr, len, prot, flags, vm_flags, 0, + &populate, NULL); + up_write(&mm->mmap_sem); + + if (populate) + mm_populate(addr, populate); + + return addr; +} + /* These take the mm semaphore themselves */ extern int __must_check vm_brk(unsigned long, unsigned long); extern int __must_check vm_brk_flags(unsigned long, unsigned long, unsigned long); From patchwork Thu Aug 30 14:38:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581847 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADD3314E1 for ; Thu, 30 Aug 2018 14:44:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C81B2B560 for ; Thu, 30 Aug 2018 14:44:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8EF132BC25; Thu, 30 Aug 2018 14:44:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ABEF82B560 for ; Thu, 30 Aug 2018 14:44:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 997526B5227; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 674726B5233; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A8D56B522A; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 97DF46B5227 for ; Thu, 30 Aug 2018 10:43:46 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id a10-v6so4010840pls.23 for ; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=NMWCOaZnHzGALwLiMCIntWS/fP3doXjqvyvXYT3P4nw=; b=CJyfiOF2q8jggNtkQlsM4nBTJYcrhkd5qrTpKkDQyNRKEPVEcMq/0gq210mNFe6F0q uuoW+rQHxY0tT1RPONJy4iJSChrpK2h/WJGGUvWvL5niRJ+W26dL0KKgxvTYdVZ3UhiM 7oyucHl4ItgBU6ubgpQflSXc7P3oEqzj7S+FRFlTuXrOU2ujRevVNw50ypfDxjYbqQkJ pzKdsPDz7LLaIwzWmOjAUbOT1z9MvlIot75o3eV9XTDuVKkriF/1DM1n7eKpHnAvKEk3 KdIWDzLlEmDty6L0/fzd+KY7coOJ94uMEv4jIzm5YdJxow4kCKhIroqcES5+3YNtkeQo e4fw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BluveZ0D5y3ElxedeQAoy8E1HPto/5+1tvDpVCH6K5pTDuAh/c KnkNadHSlCZ3d/ILD6uLhqMuwpNqsEutHDpjs+mFC2cnQ9L51+dQp9iR/pyrQCxxJZpegUO2va5 cls72AdgM4NUjhlFXIq1Y8emH3ylCMFm/JbQfKGDqEmMIxiaeK6Wep1+YsEpcilRGLQ== X-Received: by 2002:a17:902:2904:: with SMTP id g4-v6mr10311973plb.70.1535640226250; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZDadiz/eOcYa0tqzeeJ6QXeYhm9OudxldaQeVxs1eUEOnQEM22ni1rsU1QUyIENLCC3y0Q X-Received: by 2002:a17:902:2904:: with SMTP id g4-v6mr10311923plb.70.1535640225209; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=KmGPRfpkoBwB2SiEDzs7CU69JYKFN6rehLZsQRpw+G+4Tc6tkHxA0kqbdPP7POO7F8 e6S98bvlGq1sBbR77ophrszm8Zkm/MPa6GBQpEzpbYsz0hY6YzOn1RfbGLO4h/gVfph2 yBXJi5rkGsaWVkcOdam2Bo7Y/wzhYn1YHhq/pKPHWZkpsFZNbNLEWwjklRfXpAUGzrCZ ymrL/8DP/W9HrI5OXChxSHuOddD4mWlYHioxNo8DOOVKLXWU2V06EiDJQtyFV/n6Okji BjrzZ79B5XYH1BqqH6NANQMBp5BnAd2QfUx8gBLHmiV60sTExU/o0qOzPRqiSMFphrnZ 6MIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=NMWCOaZnHzGALwLiMCIntWS/fP3doXjqvyvXYT3P4nw=; b=f30YOwyE7LTOOaFZwY/dshueqSM4wtTVUPAe58DknZgBABcBf6mOqnvF1p6iKxy8Db B7iO4FNLJO72IpI5iHqB+3Wz2MQWH8Rq9AIgF4XGShwAzVcEjc9ql7d+bX83FrbWe7BW oRHMpVEqwsHiDtwAlNcmaPsth14RIotDr4fOx+3btc89t0YNmGRIRXED7dY0xcKsixTW 0mXyEGKj8T8PMSgFGM6WhuAWz1iM31Gw8ZgIg17nWZZ9pTNeLI7OP36eMPWOUyakjP5G OcX+RyJwo5oOMwX0pq/qH71gk+9Nq+4lQC5tDv4uDnaq+DcYTiLisvSGFlKYSqcz+hFQ ERsw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186724" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 18/24] x86/cet/shstk: User-mode shadow stack support Date: Thu, 30 Aug 2018 07:38:58 -0700 Message-Id: <20180830143904.3168-19-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. The shadow stack is allocated to a fixed size of RLIMIT_STACK. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 30 +++++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 14 +++ arch/x86/include/asm/processor.h | 5 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 109 +++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 24 +++++ arch/x86/kernel/process.c | 2 + fs/proc/task_mmu.c | 3 + 9 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h new file mode 100644 index 000000000000..ad278c520414 --- /dev/null +++ b/arch/x86/include/asm/cet.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; +/* + * Per-thread CET status + */ +struct cet_status { + unsigned long shstk_base; + unsigned long shstk_size; + unsigned int shstk_enabled:1; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk(void); +void cet_disable_shstk(void); +void cet_disable_free_shstk(struct task_struct *p); +#else +static inline int cet_setup_shstk(void) { return 0; } +static inline void cet_disable_shstk(void) {} +static inline void cet_disable_free_shstk(struct task_struct *p) {} +#endif + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 33833d1909af..3624a11e5ba6 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -75,7 +81,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 4731f0cf97c5..e073801a44e0 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -777,4 +777,18 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0 /* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2 /* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4 /* kernel shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7 /* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8 /* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index c24297268ebc..7be275c0b4e0 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -24,6 +24,7 @@ struct vm86; #include #include #include +#include #include #include @@ -503,6 +504,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET + struct cet_status cet; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8824d01c0c35..fbb2d91fb756 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..ec256ae27a31 --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,109 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * cet.c - Control Flow Enforcement (CET) + * + * Copyright (c) 2018, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int set_shstk_ptr(unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + return 0; +} + +static unsigned long get_shstk_addr(void) +{ + unsigned long ptr; + + if (!current->thread.cet.shstk_enabled) + return 0; + + rdmsrl(MSR_IA32_PL3_SSP, ptr); + return ptr; +} + +int cet_setup_shstk(void) +{ + unsigned long addr, size; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EOPNOTSUPP; + + size = rlimit(RLIMIT_STACK); + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + /* + * Return actual error from do_mmap(). + */ + if (addr >= TASK_SIZE_MAX) + return addr; + + set_shstk_ptr(addr + size - sizeof(u64)); + current->thread.cet.shstk_base = addr; + current->thread.cet.shstk_size = size; + current->thread.cet.shstk_enabled = 1; + return 0; +} + +void cet_disable_shstk(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return; + + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_SHSTK_EN); + wrmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, 0); + current->thread.cet.shstk_enabled = 0; +} + +void cet_disable_free_shstk(struct task_struct *tsk) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || + !tsk->thread.cet.shstk_enabled) + return; + + if (tsk == current) + cet_disable_shstk(); + + /* + * Free only when tsk is current or shares mm + * with current but has its own shstk. + */ + if (tsk->mm && (tsk->mm == current->mm) && + (tsk->thread.cet.shstk_base)) { + vm_munmap(tsk->thread.cet.shstk_base, + tsk->thread.cet.shstk_size); + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + } + + tsk->thread.cet.shstk_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 84dee5ab745a..e7eb41830add 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -411,6 +411,29 @@ static __init int setup_disable_pku(char *arg) __setup("nopku", setup_disable_pku); #endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) + cr4_set_bits(X86_CR4_CET); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk(char *s) +{ + /* require an exact match without trailing characters */ + if (strlen(s)) + return 0; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_SHSTK); + pr_info("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n"); + return 1; +} +__setup("no_cet_shstk", setup_disable_shstk); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization @@ -1375,6 +1398,7 @@ static void identify_cpu(struct cpuinfo_x86 *c) x86_init_rdrand(c); x86_init_cache_qos(c); setup_pku(c); + setup_cet(c); /* * Clear/Set all flags overridden by options, need do it diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c93fcfdf1673..4a776da4c28c 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -39,6 +39,7 @@ #include #include #include +#include /* * per-CPU TSS segments. Threads are completely 'soft' on Linux, @@ -134,6 +135,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(tsk); memset(tsk->thread.tls_array, 0, sizeof(tsk->thread.tls_array)); + cet_disable_shstk(); fpu__clear(&tsk->thread.fpu); } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5ea1d64cb0b4..b20450dde5b7 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -652,6 +652,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "", #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif }; size_t i; From patchwork Thu Aug 30 14:38:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581871 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71472174A for ; Thu, 30 Aug 2018 14:45:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 616972B560 for ; Thu, 30 Aug 2018 14:45:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 548332BC83; Thu, 30 Aug 2018 14:45:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D97672B560 for ; Thu, 30 Aug 2018 14:45:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 592EA6B522B; Thu, 30 Aug 2018 10:43:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EBB7E6B5231; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CD616B522F; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 5A7726B522A for ; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id 90-v6so4014452pla.18 for ; Thu, 30 Aug 2018 07:43:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+qTPVKzf+lhK4h/nXP6cgw6G1mAsYcJ5dN3/fmhpEfs=; b=Dvh/tsZaGxfpQV+58Ra3LRGBm9PnFhbN9PUCRQSRjl9RxGzJkQLMBgg21miFklnrD8 9ESby89OusnvJPYlDCFldGANrc9yMHi7rbd2djj5RIMW26VTzcUJd0VKKM8yfEac+kYr skF8zV5itmFszRZsVo7xHdlcXlD983ZWMpUz+CwED+qlcLpr24i+8xHi9nLKvNhDbUoB M8xnk1uXiylKf9eG+Gz3amaiJUATtZ+7W8n6b5wTb4fkDoHzCWmI8G7yBu72K23zqWsC HcZ2+Sz/YUmVauM20vgPOpkzSSHR40ijO1QNt9tnvmvyAiH752RjsDFdXwhmZm2l3Qb6 pbjQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DiwyRBCfXHd3iVZJa9qlvfJvJGZ6Wq7zwa+ltCql5bPwI3hdLI kOUPCm6hqCILuv5t9lUm0qD9u6Wb5MZ+1wX6lhFBRTNm5GPlM5z7qs5OLH880leehXRtVGYT53i NYEZEzHfmXxCKPeuf7v99q8JWYMhnnds2DN3dwecbXvWdQE0a9s8800OiJo3ilY5bCQ== X-Received: by 2002:a63:4826:: with SMTP id v38-v6mr10138928pga.379.1535640228055; Thu, 30 Aug 2018 07:43:48 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbrCDzkb+iQBsg7CoCT3DIny9ZpGRLdTKf+YJkXizeZki+KY8hgCCiG6oh9kpg4+pzc7VHf X-Received: by 2002:a63:4826:: with SMTP id v38-v6mr10138875pga.379.1535640226810; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=wifv2IU0/kv9WyQrCGlV6pNj+B4x4UJrzUa3sqIPWIw1FvT6n6qg/Xs+ZVKs4bOVTA 37DfzO9LDBLLJGER+9Cy/R5dK2AACZ4dRiGE1XbITKyxN/YOWquZTB6+9IqY/o7VHxdX 7xQLIsi/j9brg9lH5YKtxCf00iCiLyeuGvOPfRoNtK8XxAaMclB7eyCes09srNod1bx5 V6+qlJCCstPyLcAwRMR/Kc3Td1w+Giz+HgqCbLZ2gsrS5sbqv4j86DCIfxV+EgDgw88h RVh2YlVZft7G/RRVF/9snSFyb/GGFoYr7d3v5V68W0VwLEqvbjlNPzu+MTfclv7+H1B6 lRsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=+qTPVKzf+lhK4h/nXP6cgw6G1mAsYcJ5dN3/fmhpEfs=; b=INYwoUAIwF8ttuzWsKF5plqu67EGMIYZt5FBh8/Pv7OrXQoDdf/x8C1on4cLOYMabj 5j8JNyH0A1nw70fvPPQkQ6y31rnIR70hWTbwMjklAyshdo/Ii4JjVj1UIIfF+I0zia9j GjHsoWQGGRfC97UOMInsVKuQqjcjesl+H6M6zTELqam+OwHdvS4W8Q5lyY4qqLXoUpwf vLsUfkpJwTUUYlparEvUkSp0PynDm2Q53fkoJtHYFkhrYfvR/jlEiNj5WkS9TT02lpbx eLPS510Bi1UoWPkBOyI33siwbD9ZXtQoPWNcBZnJe5AsJZ4itu7uRmgw2QfbKQszRd5P Gwbw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id j11-v6si6596431pll.234.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186727" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction Date: Thu, 30 Aug 2018 07:38:59 -0700 Message-Id: <20180830143904.3168-20-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler. This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/special_insns.h | 37 ++++++++++++++++++++++++++++ arch/x86/mm/extable.c | 11 +++++++++ arch/x86/mm/fault.c | 9 +++++++ 3 files changed, 57 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 317fc59b512c..9f609e802c5c 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -237,6 +237,43 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET + +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + int err = 0; + + asm volatile("1: wrussd %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + + return err; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + BUG(); + return 0; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{ + int err = 0; + + asm volatile("1: wrussq %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + + return err; +} +#endif /* CONFIG_X86_INTEL_CET */ + #define nop() asm volatile ("nop") diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c index 45f5d6cf65ae..e06ff851b671 100644 --- a/arch/x86/mm/extable.c +++ b/arch/x86/mm/extable.c @@ -157,6 +157,17 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, } EXPORT_SYMBOL(ex_handler_clear_fs); +#ifdef CONFIG_X86_INTEL_CET +__visible bool ex_handler_wruss(const struct exception_table_entry *fixup, + struct pt_regs *regs, int trapnr) +{ + regs->ip = ex_fixup_addr(fixup); + regs->ax = -1; + return true; +} +EXPORT_SYMBOL(ex_handler_wruss); +#endif + __visible bool ex_has_fault_handler(unsigned long ip) { const struct exception_table_entry *e; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 3842353fb4a3..10dbb5c9aaef 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1305,6 +1305,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, error_code |= X86_PF_USER; flags |= FAULT_FLAG_USER; } else { + /* + * WRUSS is a kernel instrcution and but writes + * to user shadow stack. When a fault occurs, + * both X86_PF_USER and X86_PF_SHSTK are set. + * Clear X86_PF_USER here. + */ + if ((error_code & (X86_PF_USER | X86_PF_SHSTK)) == + (X86_PF_USER | X86_PF_SHSTK)) + error_code &= ~X86_PF_USER; if (regs->flags & X86_EFLAGS_IF) local_irq_enable(); } From patchwork Thu Aug 30 14:39:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 82C935A4 for ; Thu, 30 Aug 2018 14:44:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 709042B560 for ; Thu, 30 Aug 2018 14:44:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 635E02BC83; Thu, 30 Aug 2018 14:44:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75EC12B560 for ; Thu, 30 Aug 2018 14:44:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87C836B5230; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4EB576B5228; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A6746B522F; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 664B86B5226 for ; Thu, 30 Aug 2018 10:43:47 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id 132-v6so5094025pga.18 for ; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=UkpA98zALE0fynViTxAnI8knAwJHClX6QGtLN9Wtgik=; b=dXrluvpMHqGZPlHvSMREUnHdo+s2PGoGZQACPGKN3Z8adoAQuJRDmHEyTWWCyUKXGu 6bUH5LtkEizeZW18RHs+Qk0cSRIXaBjj/cgeaS+uHH/paclGWi0zyGaOjl7WotFtzm+e 96XAyOcrLXyihDhQTtlOrEsDO3NQZL/ZaSfgx5Wak2wqn78UrhCIdao/raDFoCT2vgVD FI+5Nadn+VBAdjFwlrChWAKJpWU0dQSfZvtIg8a7s4VmGa6h4PSsQbb5Q048QSKrth2y m6Q3GgDLyq30I5Cu9DE7cN7t6ODxT4BklCbGF0BhYpUoQVuwL7DnrN/7mo51fkwAFkY+ j2XQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Bp8c2rSUhxAkVXJ/ESUcTP8vJYUr0N6pIvnOil8x+dRdW5sCWz +FBNF004fHMjTYgkCq3s3d1LbcaC8EFkviiTDkW3e9unjJuGq61AxED1aqVpYYWXzEZJpvRlBFv nKjzfIJ73CHNGGhQVi3dN3dd5dI+S9FMBSrhtUpc4EwIMOgsmV1nkEaRKuRyOL90RkA== X-Received: by 2002:a62:760a:: with SMTP id r10-v6mr10884095pfc.207.1535640227071; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb3UZUmEoood1NL5Ty/s4oopXePK9ydVmGoDN5q2l/8DPDKQIthZsjxcXHXN7GmmNYR7PA3 X-Received: by 2002:a62:760a:: with SMTP id r10-v6mr10884039pfc.207.1535640226002; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=p2boVnYEeyOXenunID/k8m6jyV+xZUcITi8rCc7Ej+q2QN8HqbzowA2KIUAn9+WL0p T3QUQRGz9sdFoGvKbdqHio8E4ZpQU104T9qi8QaEOiSxzCfxJ7MLGztoCrPCyHTrs/xN F8SAAPDAPDDFt6M7ddU7SWR+IoogfvKS97MPyYiKf/jWVy1GWaJYxiSJ0QP4Fn2Twger 2ipBzxZF4TVf9DW9zz9np3MrSuV7piOStJacAD+rrkzzLmH4I6S3Vo0S8+EMzJYImoU2 FU19GKu9VPm6pifATdwcBYNQWo4uth554u3fUUq/CX/y/FOuLcN3vF2irVnbB7n8J6if Zssw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=UkpA98zALE0fynViTxAnI8knAwJHClX6QGtLN9Wtgik=; b=tEJsjmRuS/vgD2ZQ3+H7J41/L3NBmAok1ROYM0C+aIF8vLJgLRPaMgNM21c6s30oG/ yrUVmH4Nb6w7K8n4SYGIMsKKkrEMkKvqql+yek9xuw8/ToR/pcKIroKUvgcBRCFKIjb7 WnE7BRFZs8/G1QPlyKNlS4/Ns6j/p7bmA/cqreBuV8/aucjRqPzXdZOtbqQcLiiLXlo5 3LyL0Y6pWVWIkHiIuiN8HEemKXkHNhUH8Unj9rDEpJqXoakCZyUeBMErFe3tqeDL/zUf 6Zm8Hm1Bo8uMJn/CyJ5pt5rwypbtoXsurHroOK4A/BZ4ZdTP3uGwoH6TajKbi+yk//ez Utow== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id z73-v6si6610727pgd.471.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186728" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 20/24] x86/cet/shstk: Signal handling for shadow stack Date: Thu, 30 Aug 2018 07:39:00 -0700 Message-Id: <20180830143904.3168-21-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer. Signed-off-by: Yu-cheng Yu --- arch/x86/ia32/ia32_signal.c | 13 +++ arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/asm/sighandling.h | 5 ++ arch/x86/include/uapi/asm/sigcontext.h | 17 ++++ arch/x86/kernel/cet.c | 115 +++++++++++++++++++++++++ arch/x86/kernel/signal.c | 96 +++++++++++++++++++++ 6 files changed, 251 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 86b1341cba9a..cea28d2a946e 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -34,6 +34,7 @@ #include #include #include +#include /* * Do a signal return; undo the signal stack. @@ -108,6 +109,9 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, 1); + if (!err) + err = restore_sigcontext_ext(buf); + force_iret(); return err; @@ -234,6 +238,10 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, if (fpu->initialized) { unsigned long fx_aligned, math_size; + /* sigcontext extension */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, @@ -277,6 +285,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, if (ia32_setup_sigcontext(&frame->sc, fpstate, regs, set->sig[0])) return -EFAULT; + if (setup_sigcontext_ext(ksig, fpstate)) + return -EFAULT; if (_COMPAT_NSIG_WORDS > 1) { if (__copy_to_user(frame->extramask, &set->sig[1], @@ -384,6 +394,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index ad278c520414..d9ae3d86cdd7 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -19,10 +19,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); +int cet_restore_signal(unsigned long ssp); +int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} +static inline int cet_restore_signal(unsigned long ssp) { return 0; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + unsigned long *new_ssp) { return 0; } #endif #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/asm/sighandling.h b/arch/x86/include/asm/sighandling.h index bd26834724e5..23014b4082de 100644 --- a/arch/x86/include/asm/sighandling.h +++ b/arch/x86/include/asm/sighandling.h @@ -17,4 +17,9 @@ void signal_fault(struct pt_regs *regs, void __user *frame, char *where); int setup_sigcontext(struct sigcontext __user *sc, void __user *fpstate, struct pt_regs *regs, unsigned long mask); +#ifdef CONFIG_X86_64 +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fpu); +int restore_sigcontext_ext(void __user *fpu); +#endif + #endif /* _ASM_X86_SIGHANDLING_H */ diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/uapi/asm/sigcontext.h index 844d60eb1882..74f5ea5dcd24 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,23 @@ struct _xstate { /* New processor state extensions go here: */ }; +#ifdef __x86_64__ +/* + * Sigcontext extension (struct sc_ext) is located after + * sigcontext->fpstate. Because currently only the shadow + * stack pointer is saved there and the shadow stack depends + * on XSAVES, we can find sc_ext from sigcontext->fpstate. + * + * The 64-bit fpstate has a size of fpu_user_xstate_size, plus + * FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext + * is located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; +}; +#endif + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index ec256ae27a31..5cc4be6e0982 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -18,6 +18,7 @@ #include #include #include +#include static int set_shstk_ptr(unsigned long addr) { @@ -46,6 +47,69 @@ static unsigned long get_shstk_addr(void) return ptr; } +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +static int verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp = 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (ia32 && (token & 3) != 0) + return -EINVAL; + else if ((token & 3) != 1) + return -EINVAL; + + token &= ~(1UL); + + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + if ((ALIGN_DOWN(token, 8) - 8) != ssp) + return -EINVAL; + + *new_ssp = token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp = 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr = ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |= 1; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -107,3 +171,54 @@ void cet_disable_free_shstk(struct task_struct *tsk) tsk->thread.cet.shstk_enabled = 0; } + +int cet_restore_signal(unsigned long ssp) +{ + unsigned long new_ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + err = verify_rstor_token(in_ia32_syscall(), ssp, &new_ssp); + + if (err) + return err; + + return set_shstk_ptr(new_ssp); +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, + unsigned long *new_ssp) +{ + unsigned long ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + ssp = get_shstk_addr(); + err = create_rstor_token(ia32, ssp, new_ssp); + + if (err) + return err; + + if (ia32) { + ssp = *new_ssp - sizeof(u32); + err = write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp = *new_ssp - sizeof(u64); + err = write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + set_shstk_ptr(ssp); + return 0; +} diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 92a3b312a53c..e9a85689143f 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include +#include #define COPY(x) do { \ get_user_ex(regs->x, &sc->x); \ @@ -152,6 +153,10 @@ static int restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, IS_ENABLED(CONFIG_X86_32)); +#ifdef CONFIG_X86_64 + err |= restore_sigcontext_ext(buf); +#endif + force_iret(); return err; @@ -266,6 +271,11 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, } if (fpu->initialized) { +#ifdef CONFIG_X86_64 + /* sigcontext extension */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= sizeof(struct sc_ext) + 8; +#endif sp = fpu__alloc_mathframe(sp, IS_ENABLED(CONFIG_X86_32), &buf_fx, &math_size); *fpstate = (void __user *)sp; @@ -493,6 +503,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig, err |= setup_sigcontext(&frame->uc.uc_mcontext, fp, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fp); + if (err) return -EFAULT; @@ -576,6 +589,9 @@ static int x32_setup_rt_frame(struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; @@ -707,6 +723,86 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) } } +#ifdef CONFIG_X86_64 +static int copy_ext_from_user(struct sc_ext *ext, void __user *fpu) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (!access_ok(VERIFY_READ, p, sizeof(*ext))) + return -EFAULT; + + if (__copy_from_user(ext, p, sizeof(*ext))) + return -EFAULT; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + return 0; +} + +static int copy_ext_to_user(void __user *fpu, struct sc_ext *ext) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (!access_ok(VERIFY_WRITE, p, sizeof(*ext))) + return -EFAULT; + + if (__copy_to_user(p, ext, sizeof(*ext))) + return -EFAULT; + + return 0; +} + +int restore_sigcontext_ext(void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + + err = copy_ext_from_user(&ext, fp); + + if (!err) + err = cet_restore_signal(ext.ssp); + } + + return err; +} + +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + unsigned long rstor; + + rstor = (unsigned long)ksig->ka.sa.sa_restorer; + err = cet_setup_signal(is_ia32_frame(ksig), rstor, &ext.ssp); + if (!err) { + ext.total_size = sizeof(ext); + err = copy_ext_to_user(fp, &ext); + } + } + + return err; +} +#endif + static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { From patchwork Thu Aug 30 14:39:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581873 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 676035A4 for ; Thu, 30 Aug 2018 14:45:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55B462B560 for ; Thu, 30 Aug 2018 14:45:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 484D12BC83; Thu, 30 Aug 2018 14:45:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F41C2B560 for ; Thu, 30 Aug 2018 14:45:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85B316B522F; Thu, 30 Aug 2018 10:43:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 262A56B5235; Thu, 30 Aug 2018 10:43:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87AE56B5229; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 337796B5232 for ; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id c8-v6so4886513pfn.2 for ; Thu, 30 Aug 2018 07:43:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ZhdA6V38hlV1nNmtuavCJWD7maHUnpgc48DAv1CMvY4=; b=a+1kV4EE/cAEv5vK4K1Fd6Y5IzNHI0CJ+NBej8104MxZ7vQYWhYCb17zQMlP8/rs49 hXqVh9mpB+Rdcd9dpy+AV4o6MitJdfa2vXJC6uL8MmRf420RPjYoFv55fX/Y/AsuaBkm UXsqxdokRlpb5PGIlHvS8+AwWTfrfwM0xAdgPjAjoA8J0B0KKrqyLl2eXou6UxS4HSWu dwJqQc7+GRKaFQJskoD2d4n+6n8UHwhb1SHdTOnbLPdQGzs5JIDxlwE47vHlvxhbRqS2 FB6qY34UQLLaNdjKvvCeg9x01v+U/sHQRCJKrB4Zpg63zDrNydAodBMrAitu/h7I7kU9 2gVA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Dxg5ZSdQVmdOm20dwbX485YGUoDr6401C+tuVLjNqIhbEurSwt zA28na5hLJLRGc16+BUQeg56DsBWbhzcw5eFi7FEocU+XYMEe6Bmhk6vpxUoSnMgqL4HBPR0xiK qVz0nzHXru8acYO9y9xSHymFjaRivM3RZjntvtm2PV2zdLq+hSTN+MVtcn5hTU8jvPg== X-Received: by 2002:a62:1157:: with SMTP id z84-v6mr10866785pfi.66.1535640227862; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZgTB7imEsp7V2SWiqkNej7YGZiC+bGo19lWpJx49ujGbuQnAPECqVKjNt11F57ccF4aTym X-Received: by 2002:a62:1157:: with SMTP id z84-v6mr10866724pfi.66.1535640226670; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=rJqs53AQDA2GOhUljKVqKnNe+LTNoZkTt/9IWjXe20CyR0pHRRJwwbK0w62mBi3n/w rvcESvDlF1Kb+EtQIdvxoMHjDNulLk0tIK4ErVQFJjgUR0y3JbhKUtRuqaI++F9Y4haj 4gPcKNNa1A39U9NHWgBGvBzzf3sotoMIThsvhOw63qbcMqKbYYHTxbHHRxqVeYVFXX08 cK47mLkgz50k9AcNHTR6fbl6DpHXBuwBovlbF/DUFWShZj+TMZ1+yF+gZcgR/4P6MQfs 4fjcMgD0sP6jLjP3A+MdWgV8zuk/J9gnrToU2JT8ECrmOVooTLGvoH/lWtUbzSRsgLhs hr8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=ZhdA6V38hlV1nNmtuavCJWD7maHUnpgc48DAv1CMvY4=; b=omXUQdnAhRrMeeZmS0i6PdXKT0bKKmLSyZ3i1NAF9bsNfR2iFRfjENBrL/ySrE2Scj GRxOdZ9/WyoMdRptYA8INh150si5hIu2kdPt2MFm2VEK456ys+a7G16Ube3jduLPKCkm t10+ToR3lHfEJhKq/s7P9BUTcSHkfwINg4gDybRSxqkcmL65/KkmjWmCrXRaohV6X5EB QCpQCkHutcFfSCK4Iar49jOvP0AwFS6LUiB3HNQZuET0UoY7aS2GF8LN6TfBPA3OxhXA opJiglqkGF7HjSrIlB1Hb4tkraDQTTfagJKq9nunZ9gk9gdTpZo6+6JpPewUYrhnwS0G YkKg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id z73-v6si6610727pgd.471.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186730" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 21/24] x86/cet/shstk: ELF header parsing of Shadow Stack Date: Thu, 30 Aug 2018 07:39:01 -0700 Message-Id: <20180830143904.3168-22-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 4 + arch/x86/include/asm/elf.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/elf.c | 338 +++++++++++++++++++++++ fs/binfmt_elf.c | 15 + include/uapi/linux/elf.h | 1 + 7 files changed, 380 insertions(+) create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/elf.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 017b3ba70807..2cfe11e1cf7f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1919,12 +1919,16 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_PROGRAM_PROPERTIES + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n depends on CPU_SUP_INTEL && X86_64 select X86_INTEL_CET select ARCH_HAS_SHSTK + select ARCH_HAS_PROGRAM_PROPERTIES ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..5b5f169c5c07 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -382,4 +382,9 @@ struct va_alignment { extern struct va_alignment va_align; extern unsigned long align_vdso_addr(unsigned long); + +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +extern int arch_setup_features(void *ehdr, void *phdr, struct file *file, + bool interp); +#endif #endif /* _ASM_X86_ELF_H */ diff --git a/arch/x86/include/uapi/asm/elf_property.h b/arch/x86/include/uapi/asm/elf_property.h new file mode 100644 index 000000000000..af361207718c --- /dev/null +++ b/arch/x86/include/uapi/asm/elf_property.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _UAPI_ASM_X86_ELF_PROPERTY_H +#define _UAPI_ASM_X86_ELF_PROPERTY_H + +/* + * pr_type + */ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* + * Bits for GNU_PROPERTY_X86_FEATURE_1_AND + */ +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + +#endif /* _UAPI_ASM_X86_ELF_PROPERTY_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index fbb2d91fb756..36b14ef410c8 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/elf.c b/arch/x86/kernel/elf.c new file mode 100644 index 000000000000..a2c41bf39c58 --- /dev/null +++ b/arch/x86/kernel/elf.c @@ -0,0 +1,338 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Look at an ELF file's .note.gnu.property and determine if the file + * supports shadow stack and/or indirect branch tracking. + * The path from the ELF header to the note section is the following: + * elfhdr->elf_phdr->elf_note->property[]. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The .note.gnu.property layout: + * + * struct elf_note { + * u32 n_namesz; --> sizeof(n_name[]); always (4) + * u32 n_ndescsz;--> sizeof(property[]) + * u32 n_type; --> always NT_GNU_PROPERTY_TYPE_0 + * }; + * char n_name[4]; --> always 'GNU\0' + * + * struct { + * struct property_x86 { + * u32 pr_type; + * u32 pr_datasz; + * }; + * u8 pr_data[pr_datasz]; + * }[]; + */ + +#define BUF_SIZE (PAGE_SIZE / 4) + +struct property_x86 { + u32 pr_type; + u32 pr_datasz; +}; + +typedef bool (test_fn)(void *buf, u32 *arg); +typedef void *(next_fn)(void *buf, u32 *arg); + +static inline bool test_note_type_0(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + + return ((n->n_namesz == 4) && (memcmp(n + 1, "GNU", 4) == 0) && + (n->n_type == NT_GNU_PROPERTY_TYPE_0)); +} + +static inline void *next_note(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + u32 align = *arg; + int size; + + size = round_up(sizeof(*n) + n->n_namesz, align); + size = round_up(size + n->n_descsz, align); + + if (buf + size < buf) + return NULL; + else + return (buf + size); +} + +static inline bool test_property_x86(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if (pr->pr_type > max_type) + *arg = pr->pr_type; + + return (pr->pr_type == GNU_PROPERTY_X86_FEATURE_1_AND); +} + +static inline void *next_property(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if ((buf + sizeof(*pr) + pr->pr_datasz < buf) || + (pr->pr_type > GNU_PROPERTY_X86_FEATURE_1_AND) || + (pr->pr_type > max_type)) + return NULL; + else + return (buf + sizeof(*pr) + pr->pr_datasz); +} + +/* + * Scan 'buf' for a pattern; return true if found. + * *pos is the distance from the beginning of buf to where + * the searched item or the next item is located. + */ +static int scan(u8 *buf, u32 buf_size, int item_size, + test_fn test, next_fn next, u32 *arg, u32 *pos) +{ + int found = 0; + u8 *p, *max; + + max = buf + buf_size; + if (max < buf) + return 0; + + p = buf; + + while ((p + item_size < max) && (p + item_size > buf)) { + if (test(p, arg)) { + found = 1; + break; + } + + p = next(p, arg); + } + + *pos = (p + item_size <= buf) ? 0 : (u32)(p - buf); + return found; +} + +/* + * Search a NT_GNU_PROPERTY_TYPE_0 for GNU_PROPERTY_X86_FEATURE_1_AND. + */ +static int find_feature_x86(struct file *file, unsigned long desc_size, + loff_t file_offset, u8 *buf, u32 *feature) +{ + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + u32 last_pr = 0; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < desc_size; done += buf_pos) { + read_size = desc_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) + return (ret < 0) ? ret : -EIO; + + found = scan(buf, read_size, sizeof(struct property_x86), + test_property_x86, next_property, + &last_pr, &buf_pos); + + if ((!buf_pos) || found) + break; + + file_offset += buf_pos - read_size; + } + + if (found) { + struct property_x86 *pr = + (struct property_x86 *)(buf + buf_pos); + + if (pr->pr_datasz == 4) { + u32 *max = (u32 *)(buf + read_size); + u32 *data = (u32 *)((u8 *)pr + sizeof(*pr)); + + if (data + 1 <= max) { + *feature = *data; + } else { + file_offset += buf_pos - read_size; + file_offset += sizeof(*pr); + ret = kernel_read(file, feature, 4, + &file_offset); + } + } + } + + return ret; +} + +/* + * Search a PT_NOTE segment for the first NT_GNU_PROPERTY_TYPE_0. + */ +static int find_note_type_0(struct file *file, unsigned long note_size, + loff_t file_offset, u32 align, u32 *feature) +{ + u8 *buf; + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + + buf = kmalloc(BUF_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < note_size; done += buf_pos) { + read_size = note_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) { + ret = (ret < 0) ? ret : -EIO; + kfree(buf); + return ret; + } + + /* + * item_size = sizeof(struct elf_note) + elf_note.n_namesz. + * n_namesz is 4 for the note type we look for. + */ + found += scan(buf, read_size, sizeof(struct elf_note) + 4, + test_note_type_0, next_note, + &align, &buf_pos); + + file_offset += buf_pos - read_size; + + if (found == 1) { + struct elf_note *n = + (struct elf_note *)(buf + buf_pos); + u32 start = round_up(sizeof(*n) + n->n_namesz, align); + u32 total = round_up(start + n->n_descsz, align); + + ret = find_feature_x86(file, n->n_descsz, + file_offset + start, + buf, feature); + file_offset += total; + buf_pos += total; + } else if (!buf_pos) { + *feature = 0; + break; + } + } + + kfree(buf); + return ret; +} + +#ifdef CONFIG_COMPAT +static int check_notes_32(struct file *file, struct elf32_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 4)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +#ifdef CONFIG_X86_64 +static int check_notes_64(struct file *file, struct elf64_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 8)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +int arch_setup_features(void *ehdr_p, void *phdr_p, + struct file *file, bool interp) +{ + int err = 0; + u32 feature = 0; + + struct elf64_hdr *ehdr64 = ehdr_p; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return 0; + + if (ehdr64->e_ident[EI_CLASS] == ELFCLASS64) { + struct elf64_phdr *phdr64 = phdr_p; + + err = check_notes_64(file, phdr64, ehdr64->e_phnum, + &feature); + if (err < 0) + goto out; + } else { +#ifdef CONFIG_COMPAT + struct elf32_hdr *ehdr32 = ehdr_p; + + if (ehdr32->e_ident[EI_CLASS] == ELFCLASS32) { + struct elf32_phdr *phdr32 = phdr_p; + + err = check_notes_32(file, phdr32, ehdr32->e_phnum, + &feature); + if (err < 0) + goto out; + } +#endif + } + + memset(¤t->thread.cet, 0, sizeof(struct cet_status)); + + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) { + if (feature & GNU_PROPERTY_X86_FEATURE_1_SHSTK) { + err = cet_setup_shstk(); + if (err < 0) + goto out; + } + } + +out: + return err; +} diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index efae2fb0930a..b891aa292b46 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1081,6 +1081,21 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES + if (interpreter) { + retval = arch_setup_features(&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + } else { + retval = arch_setup_features(&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + + if (retval < 0) + goto out_free_dentry; +#endif + if (elf_interpreter) { unsigned long interp_map_addr = 0; diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index c5358e0ae7c5..5ef25a565e88 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -372,6 +372,7 @@ typedef struct elf64_shdr { #define NT_PRFPREG 2 #define NT_PRPSINFO 3 #define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5 #define NT_AUXV 6 /* * Note to userspace developers: size of NT_SIGINFO note may increase From patchwork Thu Aug 30 14:39:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05AED5A4 for ; Thu, 30 Aug 2018 14:46:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8B242B735 for ; Thu, 30 Aug 2018 14:46:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DC93E2BC88; Thu, 30 Aug 2018 14:46:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D1DF2B735 for ; Thu, 30 Aug 2018 14:46:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F28576B525C; Thu, 30 Aug 2018 10:45:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E85A66B525D; Thu, 30 Aug 2018 10:45:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D28016B525E; Thu, 30 Aug 2018 10:45:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 8C37B6B525C for ; Thu, 30 Aug 2018 10:45:37 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id bh1-v6so4060177plb.15 for ; Thu, 30 Aug 2018 07:45:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0wuFfnUbixdDlo99FwVTBq4Nu/kOStSfbSPh5MBldQc=; b=MFySTF0ZVh2HgUc1q/c+Ffeo/p63uu3LBxKCzqRDLAJVrb8BxGbnVeQu6TN8E89fQv hMpYUZ3385mZ1Squ30TRrTsBOeiY+2eu4VEKXZqmKYNfESQWscR8qeBQqOyr7/Rh71SK JuAdKpW8Wh8J6B03SXTW1ZxDDwWgKQEfWevfN872xchpwI8L9M/FaNQeVkFS7v9phEl4 KsybeHc47FGIi2JZDXBpLfzapmtndaMIYBrOEdbwpjwTRHxxyKRSsEhUsfl9I1x5e3tV cF49qz1IjCNfQgqA7glFb/IjcIs4/TVvD//AhrLW4A5SKPxt5D1DO0Ih6DtiqYwmDxls x7Kw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CP57vJSnCgEIz54W4WR1+DLy0jPYL1LQbqKkJmBHuCm0t3+0Wt UVS1dF6x0KbJ1+QgNV7N+pAlFcsCJWHm7VMc0Zna5Bvx8bH96C7SX8fxlHFLA/oGL9TKkqx+Ohl iU/V+Vh/eMm1ujqRksisKOUF72hzDvr3QfrLcnAKhbpfo9NEx66pG3OpF5/fGQV2fHg== X-Received: by 2002:a63:5845:: with SMTP id i5-v6mr10049645pgm.272.1535640337254; Thu, 30 Aug 2018 07:45:37 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYRGSvJOTeFpNv7JXywDRsYrzTgQGtPEtLc8zemZbqeRu+Nn2F6WQ8toL4TSSE1apAUBhd9 X-Received: by 2002:a63:5845:: with SMTP id i5-v6mr10044525pgm.272.1535640226934; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=LJXyoBiJkUPHETmZztTny4XFF/E5LYDVxVCXDk9STVZdTLeLFhO4p6PuzFVNkmA/IA +bjWnjt63H+I/EtNEqYVqGpv/8+M92Crn+YaNPm3rtXsrlo0gPLA++/BDq4hxlIRpuTj zHabf01wD+nzOZkVIDcg2dN3MuaQEa7HYedWqNiFackDfXfPtB0Kq2k0Urf3zhmNIlBD ybTMJBOVLS0Jk5uxZhAqfEWtpAXOrOIGduJDSCB2JUmfQvmwPqxGFfn+SdXIxX1p1W0M X9QdUOjadfkG5503H8K1fBH/3DU1K/rCEuq098im+S02C3Rgpph4N1cIjuY0H9Cr7ifl JxJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=0wuFfnUbixdDlo99FwVTBq4Nu/kOStSfbSPh5MBldQc=; b=R/DvNZE8S3KK1v3PH48VJFtzTsUM9SYIovhxiYGuseG6zdh7gDcIC1xdswSJJ1Xwz7 TYzanOusqhhMw1n80PgvpaWja9V9eTRZCFZ4NZ7mtWUawRHRVYlqfIE8xLzcK+uJbhAz Za0CzpqbIt4siLMkZlVhSCHopGCoO+zV685OvzNj+C8AlQswf8pGfES57w+0IXejXwjm I5OkH5iMEzRfT8uZYDiEgfc2pHsbFF6MpI4wssU1XN79wgGafVhlqYw8teEgDHR8hV3P NpYCos7P4a7ECBudPvH+H3O2ZLw5ZSrS1YkRQVUdjZbPWbYLchztl0xTbQ8iAp385fbk u8zQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id w13-v6si6403728pll.449.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186733" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 22/24] x86/cet/shstk: Handle thread shadow stack Date: Thu, 30 Aug 2018 07:39:02 -0700 Message-Id: <20180830143904.3168-23-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The shadow stack for clone/fork is handled as the following: (1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child. It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK. (2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK. (3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret. This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 34 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 ++++++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index d9ae3d86cdd7..b7b33e1026bb 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -17,12 +17,14 @@ struct cet_status { #ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk(void); +int cet_setup_thread_shstk(struct task_struct *p); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return 0; } +static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return 0; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index eeeb9289c764..8da7c999b7ee 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -223,6 +224,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 5cc4be6e0982..ce0b3b7b1160 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -134,6 +134,40 @@ int cet_setup_shstk(void) return 0; } +int cet_setup_thread_shstk(struct task_struct *tsk) +{ + unsigned long addr, size; + struct cet_user_state *state; + + if (!current->thread.cet.shstk_enabled) + return 0; + + state = get_xsave_addr(&tsk->thread.fpu.state.xsave, + XFEATURE_MASK_SHSTK_USER); + + if (!state) + return -EINVAL; + + size = tsk->thread.cet.shstk_size; + if (size == 0) + size = rlimit(RLIMIT_STACK); + + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + if (addr >= TASK_SIZE_MAX) { + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + tsk->thread.cet.shstk_enabled = 0; + return -ENOMEM; + } + + state->user_ssp = (u64)(addr + size - sizeof(u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; + return 0; +} + void cet_disable_shstk(void) { u64 r; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 4a776da4c28c..440f012ef925 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -125,6 +125,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + cet_disable_free_shstk(tsk); fpu__drop(fpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index a451bc374b9b..cfe955d8d6b2 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -317,6 +317,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp, if (sp) childregs->sp = sp; + /* Allocate a new shadow stack for pthread */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM) { + err = cet_setup_thread_shstk(p); + if (err) + goto out; + } + err = -ENOMEM; if (unlikely(test_tsk_thread_flag(me, TIF_IO_BITMAP))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr, From patchwork Thu Aug 30 14:39:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E195B174A for ; Thu, 30 Aug 2018 14:45:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D17B32B560 for ; Thu, 30 Aug 2018 14:45:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C45AF2BC83; Thu, 30 Aug 2018 14:45:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 185882B560 for ; Thu, 30 Aug 2018 14:45:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C5376B5229; Thu, 30 Aug 2018 10:43:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C95CA6B5232; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 759976B5220; Thu, 30 Aug 2018 10:43:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 2A50A6B5229 for ; Thu, 30 Aug 2018 10:43:48 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id 90-v6so4014445pla.18 for ; Thu, 30 Aug 2018 07:43:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=qpWjjwS69NtRuZcBDki/uhmcm+XjXpFLfgHPaAMpcEU=; b=iar7ztNENYoQxTP/gmb+69576n/Z5mKgPWzyi3tbxiymzWRKQDQSeFJ6e5KFLcObyg 3BdoRUsi18OdUB4k+zpRd8uT5HH/2efqd8oOvv6cQPm3bjXMKxbQO7z5bGwio8kq3IoD Y08CoEJQkjcpg7tlkmiBvtLebQWSzJKGGxEWarWZziz2/kbILV7r0ilwMKiu7FueXw7D JkXRxEbaj6yr/epGAmy4zBeJO1LQFOGAstKP7TX/7iRyewooC217fejhGVKP1LBG1tTb Xygyemf1jOUPrbxIdXQnHO6USsqLx4pU6hpEG4YN55BaBNUG5dQnBI4uZtuzuqACwiEU YMvA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Aa8B2dSvN66F9FngrnChc7cSf9KufaqW48C0D2OMTXG1Xcv02Y S0l1ee4XdULeOAO9vWzV1d1CHj89DL1Czkazwakm6zcLdPg/vzZm74yeHdNdPBrfkNuesmXIjtd YrbtYLyysOhHT8SLVBDd2I4KEAg+2LfM3dMEbX+0VOleRer2LCE3sCHoQB4OdClQ7Bg== X-Received: by 2002:a62:9992:: with SMTP id t18-v6mr10655464pfk.239.1535640227842; Thu, 30 Aug 2018 07:43:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZOJ+DSVDwabgqDhU/olIanAHec8uxxzjihUyTs+9Vm2ePDf/SjNtNiEJ5PSq+yfdlF2JVy X-Received: by 2002:a62:9992:: with SMTP id t18-v6mr10655416pfk.239.1535640226964; Thu, 30 Aug 2018 07:43:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640226; cv=none; d=google.com; s=arc-20160816; b=QP7gVJpqXSFJiukVG7kpY4KwqIVVxXkxV9mFV5I0up/CmZko+XTIKInw1nL/wQhef2 DXgLgIV6flK/8kTVwrmx+sEz9wDpcs+NwjpzlaFqSlVwvIYnMExvAdzT/HxuaXoDEt3E UHZLrM4arGCPlC8bt1nyMuV4ML6XYVcOZCUcGEjuyCaERUq6JiTEyKhPAn5EGvvf6j/w YCZW8lMMvtv8luyHj/I9LjOyL0ZK7lEp2gF8DpzaI3fAh8WyYnQq2/cMKWzAF4VivG67 nEnNWzlTzJPlwVgOfPKwzsMx2/SqY478yCixcEK0imTkQ8q2LmZEtJG9/7w5/O2I71FS eThg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=qpWjjwS69NtRuZcBDki/uhmcm+XjXpFLfgHPaAMpcEU=; b=N62CK3NkM7xA51gBoxWAnjYTM0pWOjAjlsS7oGOEQH0MwaHDOQak1BPxSIcP+4puyN uUuAnx0cq3hSa80zOMXSCNXeS2rXi4+xGO93RH7DCmKgZPh65J0LPJZkKB8ZkQyWOZFt pQ+VV4bP/uC71lW93Fmn5YpqGtzlTqABKDSvDkyR2Z+KG4jT/pqBwA8qsubF4YxfxRAr GscdaGFoBRDAG5mhQMWKfn57/TtTZjqEP8kBRfN5Nw1T2ftsabSeRQs1KMShvdN2NLHM ZHgR5RkPq/waCFDPP+Lgxf18GA6CJuM2t0fU6TuHWfFxLiYXj1jH6JkTOtBfM4GMGiV/ SSJg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id z73-v6si6610727pgd.471.2018.08.30.07.43.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:46 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186736" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:43 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 23/24] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Date: Thu, 30 Aug 2018 07:39:03 -0700 Message-Id: <20180830143904.3168-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP arch_prctl(ARCH_CET_STATUS, unsigned long *addr) Return CET feature status. The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information: *addr = SHSTK/IBT status *(addr + 1) = SHSTK base address *(addr + 2) = SHSTK size arch_prctl(ARCH_CET_DISABLE, unsigned long features) Disable CET features specified in 'features'. Return -EPERM if CET is locked. arch_prctl(ARCH_CET_LOCK) Lock in CET feature. arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr) Allocate a new SHSTK. The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 27 +++++++++++ arch/x86/kernel/cet_prctl.c | 79 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 5 ++ 6 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kernel/cet_prctl.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index b7b33e1026bb..212bd68e31d3 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -12,19 +12,24 @@ struct task_struct; struct cet_status { unsigned long shstk_base; unsigned long shstk_size; + unsigned int locked:1; unsigned int shstk_enabled:1; }; #ifdef CONFIG_X86_INTEL_CET +int prctl_cet(int option, unsigned long arg2); int cet_setup_shstk(void); int cet_setup_thread_shstk(struct task_struct *p); +int cet_alloc_shstk(unsigned long *arg); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else +static inline int prctl_cet(int option, unsigned long arg2) { return 0; } static inline int cet_setup_shstk(void) { return 0; } static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } +static inline int cet_alloc_shstk(unsigned long *arg) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return 0; } diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 5a6aac9fa41f..3aec1088e01d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -14,4 +14,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_CET_STATUS 0x3001 +#define ARCH_CET_DISABLE 0x3002 +#define ARCH_CET_LOCK 0x3003 +#define ARCH_CET_ALLOC_SHSTK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 36b14ef410c8..b9e6cdc6b4f7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,7 +139,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index ce0b3b7b1160..1c2689738604 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -110,6 +110,33 @@ static int create_rstor_token(bool ia32, unsigned long ssp, return 0; } +int cet_alloc_shstk(unsigned long *arg) +{ + unsigned long len = *arg; + unsigned long addr; + unsigned long token; + unsigned long ssp; + + addr = do_mmap_locked(0, len, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + if (addr >= TASK_SIZE_MAX) + return -ENOMEM; + + /* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + + if (!in_ia32_syscall()) + token |= 1; + ssp -= 8; + + if (write_user_shstk_64(ssp, token)) + return -EINVAL; + + *arg = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c new file mode 100644 index 000000000000..c4b7c19f5040 --- /dev/null +++ b/arch/x86/kernel/cet_prctl.c @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* See Documentation/x86/intel_cet.txt. */ + +static int handle_get_status(unsigned long arg2) +{ + unsigned int features = 0; + unsigned long shstk_base, shstk_size; + unsigned long buf[3]; + + if (current->thread.cet.shstk_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + + shstk_base = current->thread.cet.shstk_base; + shstk_size = current->thread.cet.shstk_size; + + buf[0] = (unsigned long)features; + buf[1] = shstk_base; + buf[2] = shstk_size; + return copy_to_user((unsigned long __user *)arg2, buf, + sizeof(buf)); +} + +static int handle_alloc_shstk(unsigned long arg2) +{ + int err = 0; + unsigned long shstk_size = 0; + + if (get_user(shstk_size, (unsigned long __user *)arg2)) + return -EFAULT; + + err = cet_alloc_shstk(&shstk_size); + if (err) + return err; + + if (put_user(shstk_size, (unsigned long __user *)arg2)) + return -EFAULT; + + return 0; +} + +int prctl_cet(int option, unsigned long arg2) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EINVAL; + + switch (option) { + case ARCH_CET_STATUS: + return handle_get_status(arg2); + + case ARCH_CET_DISABLE: + if (current->thread.cet.locked) + return -EPERM; + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + cet_disable_free_shstk(current); + + return 0; + + case ARCH_CET_LOCK: + current->thread.cet.locked = 1; + return 0; + + case ARCH_CET_ALLOC_SHSTK: + return handle_alloc_shstk(arg2); + + default: + return -EINVAL; + } +} diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 440f012ef925..251b8714f9a3 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -792,6 +792,11 @@ long do_arch_prctl_common(struct task_struct *task, int option, return get_cpuid_mode(); case ARCH_SET_CPUID: return set_cpuid_mode(task, cpuid_enabled); + case ARCH_CET_STATUS: + case ARCH_CET_DISABLE: + case ARCH_CET_LOCK: + case ARCH_CET_ALLOC_SHSTK: + return prctl_cet(option, cpuid_enabled); } return -EINVAL; From patchwork Thu Aug 30 14:39:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10581909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B28D5A4 for ; Thu, 30 Aug 2018 14:46:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59D602BC83 for ; Thu, 30 Aug 2018 14:46:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D3A92BEFF; Thu, 30 Aug 2018 14:46:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9AE312BC83 for ; Thu, 30 Aug 2018 14:46:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CA946B525D; Thu, 30 Aug 2018 10:45:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 231066B525F; Thu, 30 Aug 2018 10:45:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 030BB6B5260; Thu, 30 Aug 2018 10:45:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id AE4146B525D for ; Thu, 30 Aug 2018 10:45:47 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id b6-v6so4051892pls.16 for ; Thu, 30 Aug 2018 07:45:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=BOvvh/3hcu+naO0uB8W/LFgIdtQUZcztYJHbYiW1fqqGWqCG7LCJaa3QY+jBUZVCWo OQpuqn31sICv2pP+U5XfYgjQLGOxMgK/JgluGndf0ROUp/5qEmbZffoL4OR2GlTbSJRj 0dRWUF1dwm9QsTLrgtbi9xoYiiY0mVTSY6wXJ3dZjbO0ofaeA+y0GhZOy+LJtOLk0JEf TSItTOZ0izi59WuZj8ibzzrDC4+m7CqCtmMyZJjNEo8hL9zUcuUxhrOdaGk8BUO6NWYN 7TZDOTqzgsDZ27IfpzjW/4TEMv8LMWew7DIh3jBr4Hw9tKL9wdrvTX0shXqu5au6rz3R Aixg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51D22SkVqq1hjExwRtUuUZSOE+cENsVLrrDlX0msKc4jK40IIgn0 XGdeYwLBWu58ldtlJIQeTvBz85Kz4O6WA7XnaStwRu18ZSz9GNLAIdKW/6HZOAf+R8oDXT1zLj8 KdoV0AVZ00xoywvT93iPIZAo8FhaXV4ocK6qHzEAwXnZ4qYgWIf7foyju7a6dq3Ncww== X-Received: by 2002:a63:7a45:: with SMTP id j5-v6mr9652266pgn.363.1535640347336; Thu, 30 Aug 2018 07:45:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZ8FKawqG5e0+TNeakB6f3rI8VGop9VP4IUnYpcYlIAsvcj1J3hlN5pZcpFxR+K4zB6BEBn X-Received: by 2002:a63:7a45:: with SMTP id j5-v6mr9646968pgn.363.1535640225409; Thu, 30 Aug 2018 07:43:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535640225; cv=none; d=google.com; s=arc-20160816; b=J9c0InCvvwjZKVxyzBToMFd0J9CvewsTHOYesdH34PW6aQnQ3EPF+Hn/gotNiY19gY 1YN0gN42d3OxF+hLLkYUC2q7yj1CTpX1JZ48YWA369pbmGLLgidSmxqAOQi/bYKsBMaG hlCndplPvm0qIpI0PtVAEC3m+qjLJTlu2WfH7TtEsM1YgK7lbRXfuTcGSOQG3JKztdux Spgz/V+ebG56evcHpSmDDtMFApVVSYzcnO0iomzn2kaNIv7sr9BA4xZBsdjGmxYL1ffK Ib2dQYR0aQ/Il5J2OL4XN56NGS1it18VZyC9ZKsTiIMABi5kDzFA3r2GohHCIZsqKd+8 4Zfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=okFnB9GHw6qIUFNsaJa2JTRBBVn09ECLQiPXm14mCYO0//tYYRji2jc9QTw032Tf3v 8i0lDuxMPVazr5gShJAFSmspCWshVavLgILL6ySy+x+vwUWqah+OQnY8eMNNWfTJ7ZX1 V2llrnYMboviGg1YZIRZtvfmDw8smBLpWVD1xwBE3gR0aC4ERGKTxvRZNPAggH2BJK3g fURxSDt6gISPldd5nU/hDrLblMKG0ISLfeDs5rDkMDI8rvSwE3ghTk9ewjZr0U9QRsSE aA3MZgr+MwSoGW1RbbOGdzEiP+u5KU8Jpbd22/9unL92RfYZRprVHU3cmZxJy9Mimpby QzPQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id c15-v6si6092534plo.232.2018.08.30.07.43.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:43:45 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Aug 2018 07:43:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,307,1531810800"; d="scan'208";a="67186742" Received: from 2b52.sc.intel.com ([143.183.136.52]) by fmsmga008.fm.intel.com with ESMTP; 30 Aug 2018 07:43:44 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v3 24/24] x86/cet/shstk: Add Shadow Stack instructions to opcode map Date: Thu, 30 Aug 2018 07:39:04 -0700 Message-Id: <20180830143904.3168-25-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180830143904.3168-1-yu-cheng.yu@intel.com> References: <20180830143904.3168-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the following shadow stack management instructions. INCSSP: Increment shadow stack pointer by the steps specified. RDSSP: Read SSP register into a GPR. SAVEPREVSSP: Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack. RSTORSSP: Restore from a "restore token" pointed by a GPR to SSP. WRSS: Write to kernel-mode shadow stack (kernel-mode instruction). WRUSS: Write to user-mode shadow stack (kernel-mode instruction). SETSSBSY: Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR. CLRSSBSY: Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable