From patchwork Fri Sep 21 15:03:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610217 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C10C25A4 for ; Fri, 21 Sep 2018 15:08:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF96A2E421 for ; Fri, 21 Sep 2018 15:08:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A2E482E423; Fri, 21 Sep 2018 15:08:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 174812E421 for ; Fri, 21 Sep 2018 15:08:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2110F8E0009; Fri, 21 Sep 2018 11:08:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1978D8E0002; Fri, 21 Sep 2018 11:08:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 085A58E0008; Fri, 21 Sep 2018 11:08:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id D1E5D8E0002 for ; Fri, 21 Sep 2018 11:08:49 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id p22-v6so6677835pfj.7 for ; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GItZ6zMjewsshyErngy5TzAWyns2xf+Aa7z7tr61mP8=; b=X03cW1SKHKCLCCVjLXDRKWKyOYl0rlsm7CXr+zH/NbDW0jSZHeykVqfx8GEs8UzU/r qvnUMIk8I2EmhdKoctC0cxlXkedwdVgJ7L+8DFTGSMUzTVLRNz7fRe6ejLYAeTOSU0uA B6QwMMAI74nlP/V3j2XFzW7Hm7W9SdL/tM/MSUDWiwXhV47yuPRfUpeNV8DKMQMEQopv GqX6uE/zJRgzl6bw0churdBTHNDZ9Xk3IF2DywyDFQv5OBxlexKfysaaFUW/TbxQapN2 HJ3uBaqYUYIslGQPgJ8lGyAhFDMSRuJzJJJ1U1lFHYjvic7WuHfwtaQ0j1RLIWTiH3cO Ei2g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Dl28nGL9RB9Vtd0lmWKtUJR47mhHddUQ4hf7bW6U+Y7W4GFyUz PX32sq3Fy1F34LjQkhpdkxQDwYViD8UTVZLPf3iXjir14AKAfq9f9tr+R5you4WjWx57t76O3i4 VawfLlka0IN2Vs0R4XsUYCRw8CSwj6ajmpdffNQtgkdIn2AXOPgN8Pu1+Ml7/yz7qow== X-Received: by 2002:a63:4a09:: with SMTP id x9-v6mr40440126pga.34.1537542529513; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZzfJShXDDFvp0nlRiE7rBrJtzomxuBY8bSMCru33X+zSuzYDO869w39IwM6srDIHEKlw/D X-Received: by 2002:a63:4a09:: with SMTP id x9-v6mr40440073pga.34.1537542528504; Fri, 21 Sep 2018 08:08:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542528; cv=none; d=google.com; s=arc-20160816; b=d21nwx2jrkIQUXz8ignmMkB3N9LR6ewPbYer6NppbRmcmwxbJj+a1KCoZBX6Ic3GoN UAdykzNAQq0gwSKOTYk7UANmhIXbU2ZB0p7cO4YRtq0ciL8TDSpHZLoFzAxFGqtQXvXd 2ULR0KM52Ip6BLYEeoM6iQPWZkTS6X5Si17GXC4IEBGzt81rMCbF9tjo0NOHo3dz6UXC hCR4QgdqiwdRLpC4oQgB9+1IjrXWG2vuACOic2+Lqfjr32CSzwIoKYCE6wUR9vql0jUg dw08HNXXgo+rtFdlW2PcHGPg3R3X1GVLFs2E+YQw1yQgYzz6h0nQ8VAX2R+7gOG0mPH3 PACA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=GItZ6zMjewsshyErngy5TzAWyns2xf+Aa7z7tr61mP8=; b=EZx1v2Xu9zMmRejcATEklt/EjaugxaujF86saJLqjkmfq6Ah1jiKSb3sTA8DJFywYI qcYsxOCs7Nf3hhh9zZ2HkjN3ZKmqTYwyXiOC5KYuWF4ZBAV5Eoez7GW7hJHrJjiFW01h k05gsbCrjA2e5AiZtOYHdXNYMnowZgUx1/fV0TjBNITEYSpOqz/YdOvafN+D+qSqvKin Rd9fQ5ILQx9Fl/A2B3eZhZNLs01HagBfRT8vLMSwjiLVTo0iQnkcv+DPj1SlhqZzSxQF uFjyx3NOBWqkS+cwg1w3pxCPqOxsgcGCc0T+h/7lq/uN1qDfHhCVKe73G4RbZHhrQT1z dDFA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:48 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856530" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 01/27] x86/cpufeatures: Add CPUIDs for Control-flow Enforcement Technology (CET) Date: Fri, 21 Sep 2018 08:03:25 -0700 Message-Id: <20180921150351.20898-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add CPUIDs for Control-flow Enforcement Technology (CET). CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 3 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 89a048c2faec..fa69651a017e 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -221,6 +221,7 @@ #define X86_FEATURE_ZEN ( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */ #define X86_FEATURE_L1TF_PTEINV ( 7*32+29) /* "" L1TF workaround PTE inversion */ #define X86_FEATURE_IBRS_ENHANCED ( 7*32+30) /* Enhanced IBRS */ +#define X86_FEATURE_IBT ( 7*32+31) /* Indirect Branch Tracking */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */ @@ -321,6 +322,7 @@ #define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c index 772c219b6889..63cbb4d9938e 100644 --- a/arch/x86/kernel/cpu/scattered.c +++ b/arch/x86/kernel/cpu/scattered.c @@ -21,6 +21,7 @@ struct cpuid_bit { static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_APERFMPERF, CPUID_ECX, 0, 0x00000006, 0 }, { X86_FEATURE_EPB, CPUID_ECX, 3, 0x00000006, 0 }, + { X86_FEATURE_IBT, CPUID_EDX, 20, 0x00000007, 0}, { X86_FEATURE_CAT_L3, CPUID_EBX, 1, 0x00000010, 0 }, { X86_FEATURE_CAT_L2, CPUID_EBX, 2, 0x00000010, 0 }, { X86_FEATURE_CDP_L3, CPUID_ECX, 2, 0x00000010, 1 }, From patchwork Fri Sep 21 15:03:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610237 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 358325A4 for ; Fri, 21 Sep 2018 15:09:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2422A2E421 for ; Fri, 21 Sep 2018 15:09:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 174D92E426; Fri, 21 Sep 2018 15:09:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E70122E421 for ; Fri, 21 Sep 2018 15:09:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5E058E0001; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 78F4E8E000B; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14E1A8E0015; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 6A6228E000E for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id g12-v6so6322881plo.1 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=kYlQ1ATYPX18BODPZ6MDhmQ3OmDqE0vEbn/QaIqn05k=; b=Ki3THU+9LpN3kBGOr6f1gf8lGcbzn46POxbxC9ve6EL5eWqmiX40RgPU0DrRnE7+/X Rpd+fF2uPfo1VFb8ldkCKBrP4n2z4TsPSpZ+Yi05LmP6F1KVQ4EL+QzxPLHfFHheie1N oFZoPUrGboE1CjSUS/LNeoKzSpzXbxz4Hy6wxw0SnpZxLBwLeD5z3usR+5RDHahyMHe4 lRwenI7HmVKRu3pR1mfir+p5adCXhIZl7PMmYu23Gr9vYJUwY5kli8wfUC0wiQ+3ZNkD N52iqXW5kQTzuIo+I9Yers5TGepGxFj0x7LrZOy+odTV8f5SdnbZiEKR11agc2LS18Qk v6MQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CnvXqT63pfPy+4sRfoEGum9Su4tNAh1y1zsgIWQCUAFz1HyWhS Uof6UL8HANe9AyTvpOsvH2BxHzWq9wGTws9lz1R+aD0NLG4YlAIAm50bMNoldV5IdQ2H/qa83Jr ndrjCu8zIshe6cCHQ0a4/Xanpp6qxPGLwlUYkcM1KwPz/RUgNVZvyfI+fhzdrz5vOog== X-Received: by 2002:a62:6948:: with SMTP id e69-v6mr46789423pfc.166.1537542531057; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdY70M9nIvWJyMZR3nOt/bJoV1O0QbNGlLrAxOpyQW7cEH6nI7teTIFVk+++Fq5VrGsF/CYI X-Received: by 2002:a62:6948:: with SMTP id e69-v6mr46789277pfc.166.1537542528830; Fri, 21 Sep 2018 08:08:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542528; cv=none; d=google.com; s=arc-20160816; b=o/7OgRZ38OUnUhO8qy6AY0mo8bgdAoRvyn+wsySsazkltnH3X/ji/SVxxCD0+8gDWe CRqMFjyZmHhBYr8nk4+puejcmaOgt2m9NdRbN18fRQb5COcr03K36z/YeFMAL3L7/n1m t3YH85BjK1BU7mpZGJtnY0sywGzMNyEkjEcOZJqc1l264HBg+2tMFGBb35CgjQZV+hZU 243jY7GpOzkzUi6EW8g+4zumNPdKdbOk1RLrHJOQn7L3bbXufg/5zPtvXyZj8/yAwMFN 9/0PQnBSZuTwyl5fJvW57h13pzzGXYRI+YWJr0ONjZzJ8U3kW4YzszJQGpHRnrYl80Tc R+7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=kYlQ1ATYPX18BODPZ6MDhmQ3OmDqE0vEbn/QaIqn05k=; b=RYgM5wS+EbL5f0c6avWbccu1GTWn5QTsvEESdOOiRPiTWG5YKtfdswj+HwvY1i5Iel /VmbPuLDoxFkc7S5pDx9K9yxZu0IEC4USDysfXmDYRTkapRZO63Cb3nGWvtuMPoMpq9f Q6WEbFTL2azHkYGd4gGDvhTDzeAYAgD7E1MvJMlTt1IRVgllXi3m4kupf7cz4cSHf0RC 3yu+vs2KkQ76PDu407k9UyP1xqH7vy09jRT7VzRm4VXUSdsyuWQjqHXWcSTiK/FZLcRB djpU3p0NF8FPwT9tmx9uicbNhvy4yshiTW9H2kAEOB8T6UWENsGmN5dGEn8zvn2sXfT0 zcnA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:48 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856534" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 02/27] x86/fpu/xstate: Change some names to separate XSAVES system and user states Date: Fri, 21 Sep 2018 08:03:26 -0700 Message-Id: <20180921150351.20898-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To support XSAVES system states, change some names to distinguish user and system states. Change: supervisor to system copy_init_fpstate_to_fpregs() to copy_init_user_fpstate_to_fpregs() xfeatures_mask to xfeatures_mask_user XCNTXT_MASK to SUPPORTED_XFEATURES_MASK (states supported) Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 24 ++++---- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +- arch/x86/kernel/fpu/xstate.c | 88 +++++++++++++++-------------- 6 files changed, 66 insertions(+), 63 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index a38bf5a1e37a..f1f9bf91a0ab 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -93,7 +93,8 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | + xfeatures_mask_user; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -233,7 +234,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) /* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact - * format and supervisor states in addition to modified optimization in + * format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 48581988d78c..9b382e5157ed 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,19 +19,19 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) +/* System features */ +#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT) /* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " @@ -39,7 +39,7 @@ #define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 2ea85b32421a..4bd56079048f 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -363,7 +363,7 @@ void fpu__drop(struct fpu *fpu) * Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs(void) +static inline void copy_init_user_fpstate_to_fpregs(void) { if (use_xsave()) copy_kernel_to_xregs(&init_fpstate.xsave, -1); @@ -395,7 +395,7 @@ void fpu__clear(struct fpu *fpu) preempt_disable(); fpu__initialize(fpu); user_fpu_begin(); - copy_init_fpstate_to_fpregs(); + copy_init_user_fpstate_to_fpregs(); preempt_enable(); } } diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 6abd83572b01..761c3a5a9e07 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -229,7 +229,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ u64 __init fpu__get_supported_xfeatures_mask(void) { - return XCNTXT_MASK; + return SUPPORTED_XFEATURES_MASK; } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 23f1691670b6..f77aa76ba675 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -249,11 +249,11 @@ static inline int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_ { if (use_xsave()) { if ((unsigned long)buf % 64 || fx_only) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_fxregs(buf); } else { - u64 init_bv = xfeatures_mask & ~xbv; + u64 init_bv = xfeatures_mask_user & ~xbv; if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_xregs(buf, xbv); @@ -417,7 +417,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 87a57b7642d3..19f8df54c72a 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -53,11 +53,11 @@ static short xsave_cpuid_features[] __initdata = { /* * Mask of xstate features supported by the CPU and the kernel: */ -u64 xfeatures_mask __read_mostly; +u64 xfeatures_mask_user __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +82,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -113,14 +113,14 @@ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) } EXPORT_SYMBOL_GPL(cpu_has_xfeatures); -static int xfeature_is_supervisor(int xfeature_nr) +static int xfeature_is_system(int xfeature_nr) { /* - * We currently do not support supervisor states, but if + * We currently do not support system states, but if * we did, we could find out like this. * * SDM says: If state component 'i' is a user state component, - * ECX[0] return 0; if state component i is a supervisor + * ECX[0] return 0; if state component i is a system * state component, ECX[0] returns 1. */ u32 eax, ebx, ecx, edx; @@ -131,7 +131,7 @@ static int xfeature_is_supervisor(int xfeature_nr) static int xfeature_is_user(int xfeature_nr) { - return !xfeature_is_supervisor(xfeature_nr); + return !xfeature_is_system(xfeature_nr); } /* @@ -164,7 +164,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask) == xfeatures_mask) + if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) return; /* @@ -191,7 +191,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * in a special way already: */ feature_bit = 0x2; - xfeatures = (xfeatures_mask & ~xfeatures) >> 2; + xfeatures = (xfeatures_mask_user & ~xfeatures) >> 2; /* * Update all the remaining memory layouts according to their @@ -219,20 +219,20 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) return; /* - * Make it clear that XSAVES supervisor states are not yet + * Make it clear that XSAVES system states are not yet * implemented should anyone expect it to work by changing * bits in XFEATURE_MASK_* macros and XCR0. */ - WARN_ONCE((xfeatures_mask & XFEATURE_MASK_SUPERVISOR), - "x86/fpu: XSAVES supervisor states are not yet implemented.\n"); + WARN_ONCE((xfeatures_mask_user & XFEATURE_MASK_SYSTEM), + "x86/fpu: XSAVES system states are not yet implemented.\n"); - xfeatures_mask &= ~XFEATURE_MASK_SUPERVISOR; + xfeatures_mask_user &= ~XFEATURE_MASK_SYSTEM; cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -242,7 +242,7 @@ void fpu__init_cpu_xstate(void) */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask & (1UL << xfeature)); + return !!(xfeatures_mask_user & BIT_ULL(xfeature)); } /* @@ -272,7 +272,7 @@ static void __init setup_xstate_features(void) cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx); /* - * If an xfeature is supervisor state, the offset + * If an xfeature is system state, the offset * in EBX is invalid. We leave it to -1. */ if (xfeature_is_user(i)) @@ -348,7 +348,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; int i; /* @@ -421,7 +421,8 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = (u64)1 << 63 | xfeatures_mask; + init_fpstate.xsave.header.xcomp_bv = + BIT_ULL(63) | xfeatures_mask_user; /* * Init all the features state with header.xfeatures being 0x0 @@ -440,11 +441,11 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports supervisor states and it uses compacted - * format. Checking a supervisor state's uncompacted offset is + * Only XSAVES supports system states and it uses compacted + * format. Checking a system state's uncompacted offset is * an error. */ - if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { + if (XFEATURE_MASK_SYSTEM & (1 << xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -465,7 +466,7 @@ static int xfeature_size(int xfeature_nr) /* * 'XSAVES' implies two different things: - * 1. saving of supervisor/system state + * 1. saving of system state * 2. using the compacted format * * Use this function when dealing with the compacted format so @@ -480,8 +481,8 @@ int using_compacted_format(void) /* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ int validate_xstate_header(const struct xstate_header *hdr) { - /* No unknown or supervisor features may be set */ - if (hdr->xfeatures & (~xfeatures_mask | XFEATURE_MASK_SUPERVISOR)) + /* No unknown or system features may be set */ + if (hdr->xfeatures & (~xfeatures_mask_user | XFEATURE_MASK_SYSTEM)) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -588,11 +589,11 @@ static void do_extra_xstate_size_checks(void) check_xstate_against_struct(i); /* - * Supervisor state components can be managed only by + * System state components can be managed only by * XSAVES, which is compacted-format only. */ if (!using_compacted_format()) - XSTATE_WARN_ON(xfeature_is_supervisor(i)); + XSTATE_WARN_ON(xfeature_is_system(i)); /* Align from the end of the previous feature */ if (xfeature_is_aligned(i)) @@ -616,7 +617,7 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask. + * Get total size of enabled xstates in XCR0/xfeatures_mask_user. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer @@ -706,7 +707,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { - xfeatures_mask = 0; + xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); } @@ -742,15 +743,15 @@ void __init fpu__init_system_xstate(void) } cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask = eax + ((u64)edx << 32); + xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue * booting without it. This is too early to BUG(). */ - pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask); + pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask_user); goto out_disable; } @@ -759,10 +760,10 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask &= ~BIT(i); + xfeatures_mask_user &= ~BIT_ULL(i); } - xfeatures_mask &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -772,9 +773,10 @@ void __init fpu__init_system_xstate(void) /* * Update info used for ptrace frames; use standard-format size and no - * supervisor xstates: + * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, + xfeatures_mask_user & ~XFEATURE_MASK_SYSTEM); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -782,7 +784,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask, + xfeatures_mask_user, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -801,7 +803,7 @@ void fpu__resume_cpu(void) * Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has(X86_FEATURE_XSAVE)) - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -853,7 +855,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) * have not enabled. Remember that pcntxt_mask is * what we write to the XCR0 register. */ - WARN_ONCE(!(xfeatures_mask & xstate_feature), + WARN_ONCE(!(xfeatures_mask_user & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -1003,7 +1005,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= ~XFEATURE_MASK_SYSTEM; /* * Copy xregs_state->header: @@ -1087,7 +1089,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= ~XFEATURE_MASK_SYSTEM; /* * Copy xregs_state->header: @@ -1180,7 +1182,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; /* * Add back in the features that came in from userspace: @@ -1236,7 +1238,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; /* * Add back in the features that came in from userspace: From patchwork Fri Sep 21 15:03:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610229 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 26CFE15A6 for ; Fri, 21 Sep 2018 15:09:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15F7D2E421 for ; Fri, 21 Sep 2018 15:09:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 099AD2E426; Fri, 21 Sep 2018 15:09:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F1C7C2E421 for ; Fri, 21 Sep 2018 15:09:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 792FF8E000E; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5B5378E000C; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE42F8E0012; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 2F23E8E000B for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id j15-v6so6656700pff.12 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9YKfHw/0dgglNTt5KC6AtpZS7C4X66pTo/aEPMbL4Vc=; b=rfvEoP8utcOl4tDyKnrahyqQKLaU4Zb2kho+e2drduP/9olYec2kMc9beyQTY/g7Ie wlEiguLb1uiTJldu7Pat1cinH7viccsZd7WXmi2vqE2HTTLnHqlAWNVGmUiwchl1IfTk gP5mIi92hXx6U0VRpAyCTgIuAjiSnJzRnCV2JCpcC145NuRFsV0UBw57B2EAG6hIwrQP sVlrDHo2Ot94jqzs6Dpsw5rI6Q6zIDTQ7GL8aSnn1ExJXD+/HCath0Z7lv3O8IM+kn9W EyXNUoHfQ8HpWjpqBi0kQNZIMFplEXxD460fAcQWqXMr+HL3u2r0C0MTR5+RTstN3+k9 PCNg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AiLOhP18ibS88dVcRZRyt2/5hWkCsfcvU8vF1BEZJNltxM9kGA aVVzPNHFT1nujtmH0U8pAHFSQH1Zxpz6HcVopnip6fC1KxQilh/S2FBd/kN+k5DLjeadrPFU9qE c7X7QY6TIZhloNszfJ6xX491vgxkYtEEP/qtiNMNXg9r+sjFdefgO5wvS5FMK8zNAaA== X-Received: by 2002:a63:fc07:: with SMTP id j7-v6mr41656218pgi.1.1537542530806; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdaf6MrcZMjxrQrd1cDnq7c++/FTN89jc60ROxTFlfLCPJykAI1/FHxX1koBVmZw9TnOFekf X-Received: by 2002:a63:fc07:: with SMTP id j7-v6mr41656122pgi.1.1537542529188; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=n1hkKs7LKVev7H6cc92Zkgi3ya1BaFfgYsVTNS5jjWquIOJn0Z/LPNAtC/mpyJZU9r mSt5He9ekhO4H7ObYz386V3rni7jQ8TZks8oBDHHtNJpXQ5y87M9TnfNyLLyz4/4vw4r MIEJSNgnnYDRdDzn3bUWLZXSGc1eCPdHDuny0qXLF6Yz1uRFlyzewId/Zx0lFoT13y34 sJN7ayg9Apr06snZw/E+8/K8uxbdAUrUqPwc9AZJ5Gry12APXLGpwIqwDTBObZzAJsI/ a02avVGUGIeNc1blPH71mgt6iGnVS0GW+MoDeOxh031Qmcq6OprAZH1skZ3zysExC76U ijkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=9YKfHw/0dgglNTt5KC6AtpZS7C4X66pTo/aEPMbL4Vc=; b=EHmnHa+PuWvtFWfO8phsR+z60NdRPdcTOPp5kQ6dSXKk2Qc7m+HAtzA7GWch9iI6eL BBAIbeivdMX1gV4EuodVs7Am31a2KT6HFlx3jeemjTwYva1+43EasFCXt+wIijxEtrVr P4WgPSSxfoWzE7rgyVsW/qpuKaP0WaNjsPtjwg48qkDVYozAzkhJ65Luu/elFX1x61uS /VXyLDG5Enln/uJSevPVLgAASWH2MZ0WA3jNpwtXuBMIqYH5Y1vUQAJZOatSpFBZTGMr djV2K0dIvrnULce78b8JZxUEL5MFsEL+HIjqWwsV2lzON/CGGuwUwnyvOZcSanfLAffT FQCw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856536" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 03/27] x86/fpu/xstate: Enable XSAVES system states Date: Fri, 21 Sep 2018 08:03:27 -0700 Message-Id: <20180921150351.20898-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP XSAVES saves both system and user states. The Linux kernel currently does not save/restore any system states. This patch creates the framework for supporting system states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 3 +- arch/x86/include/asm/fpu/xstate.h | 9 ++- arch/x86/kernel/fpu/core.c | 7 +- arch/x86/kernel/fpu/init.c | 10 --- arch/x86/kernel/fpu/xstate.c | 112 +++++++++++++++++----------- 5 files changed, 80 insertions(+), 61 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index f1f9bf91a0ab..1f447865db3a 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -45,7 +45,6 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system(struct cpuinfo_x86 *c); extern void fpu__init_check_bugs(void); extern void fpu__resume_cpu(void); -extern u64 fpu__get_supported_xfeatures_mask(void); /* * Debugging facility: @@ -94,7 +93,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * trigger #GP: */ xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | - xfeatures_mask_user; + xfeatures_mask_all; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 9b382e5157ed..a32dc5f8c963 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,10 +19,10 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* System features */ -#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT) - -/* All currently supported features */ +/* + * SUPPORTED_XFEATURES_MASK indicates all features + * implemented in and supported by the kernel. + */ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ XFEATURE_MASK_YMM | \ @@ -40,6 +40,7 @@ #endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 4bd56079048f..9f51b0e1da25 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -365,8 +365,13 @@ void fpu__drop(struct fpu *fpu) */ static inline void copy_init_user_fpstate_to_fpregs(void) { + /* + * Only XSAVES user states are copied. + * System states are preserved. + */ if (use_xsave()) - copy_kernel_to_xregs(&init_fpstate.xsave, -1); + copy_kernel_to_xregs(&init_fpstate.xsave, + xfeatures_mask_user); else if (static_cpu_has(X86_FEATURE_FXSR)) copy_kernel_to_fxregs(&init_fpstate.fxsave); else diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 761c3a5a9e07..eaf9d9d479a5 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -222,16 +222,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* - * Find supported xfeatures based on cpu features and command-line input. - * This must be called after fpu__init_parse_early_param() is called and - * xfeatures_mask is enumerated. - */ -u64 __init fpu__get_supported_xfeatures_mask(void) -{ - return SUPPORTED_XFEATURES_MASK; -} - /* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch(void) { diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 19f8df54c72a..dd2c561c4544 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -51,13 +51,16 @@ static short xsave_cpuid_features[] __initdata = { }; /* - * Mask of xstate features supported by the CPU and the kernel: + * Mask of xstate features supported by the CPU and the kernel. + * This is the result from CPUID query, SUPPORTED_XFEATURES_MASK, + * and boot_cpu_has(). */ u64 xfeatures_mask_user __read_mostly; +u64 xfeatures_mask_all __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_all)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +85,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -164,7 +167,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) + if ((xfeatures & xfeatures_mask_all) == xfeatures_mask_all) return; /* @@ -219,30 +222,31 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_all) return; + + cr4_set_bits(X86_CR4_OSXSAVE); + /* - * Make it clear that XSAVES system states are not yet - * implemented should anyone expect it to work by changing - * bits in XFEATURE_MASK_* macros and XCR0. + * XCR_XFEATURE_ENABLED_MASK sets the features that are managed + * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be + * set here. */ - WARN_ONCE((xfeatures_mask_user & XFEATURE_MASK_SYSTEM), - "x86/fpu: XSAVES system states are not yet implemented.\n"); + xsetbv(XCR_XFEATURE_ENABLED_MASK, + xfeatures_mask_user); - xfeatures_mask_user &= ~XFEATURE_MASK_SYSTEM; - - cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + /* + * MSR_IA32_XSS sets which XSAVES system states to be managed by + * XSAVES. Only XSAVES system states can be set here. + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, + xfeatures_mask_all & ~xfeatures_mask_user); } -/* - * Note that in the future we will likely need a pair of - * functions here: one for user xstates and the other for - * system xstates. For now, they are the same. - */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask_user & BIT_ULL(xfeature)); + return !!(xfeatures_mask_all & BIT_ULL(xfeature)); } /* @@ -348,7 +352,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_all)*8]; int i; /* @@ -422,7 +426,7 @@ static void __init setup_init_fpu_buf(void) if (boot_cpu_has(X86_FEATURE_XSAVES)) init_fpstate.xsave.header.xcomp_bv = - BIT_ULL(63) | xfeatures_mask_user; + BIT_ULL(63) | xfeatures_mask_all; /* * Init all the features state with header.xfeatures being 0x0 @@ -441,11 +445,10 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports system states and it uses compacted - * format. Checking a system state's uncompacted offset is - * an error. + * Checking a system or unsupported state's uncompacted offset + * is an error. */ - if (XFEATURE_MASK_SYSTEM & (1 << xfeature_nr)) { + if (~xfeatures_mask_user & BIT_ULL(xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -482,7 +485,7 @@ int using_compacted_format(void) int validate_xstate_header(const struct xstate_header *hdr) { /* No unknown or system features may be set */ - if (hdr->xfeatures & (~xfeatures_mask_user | XFEATURE_MASK_SYSTEM)) + if (hdr->xfeatures & ~xfeatures_mask_user) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -617,15 +620,12 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask_user. + * Get total size of enabled xstates in XCR0 | IA32_XSS. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer * that we use 'XSAVES' on, we could potentially overflow the * buffer because 'XSAVES' saves system states too. - * - * Note that we do not currently set any bits on IA32_XSS so - * 'XCR0 | IA32_XSS == XCR0' for now. */ static unsigned int __init get_xsaves_size(void) { @@ -707,6 +707,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { + xfeatures_mask_all = 0; xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); @@ -722,6 +723,8 @@ void __init fpu__init_system_xstate(void) static int on_boot_cpu __initdata = 1; int err; int i; + u64 cpu_user_xfeatures_mask; + u64 cpu_system_xfeatures_mask; WARN_ON_FPU(!on_boot_cpu); on_boot_cpu = 0; @@ -742,10 +745,24 @@ void __init fpu__init_system_xstate(void) return; } + /* + * Find user states supported by the processor. + * Only these bits can be set in XCR0. + */ cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask_user = eax + ((u64)edx << 32); + cpu_user_xfeatures_mask = eax + ((u64)edx << 32); + + /* + * Find system states supported by the processor. + * Only these bits can be set in IA32_XSS MSR. + */ + cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx); + cpu_system_xfeatures_mask = ecx + ((u64)edx << 32); - if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + xfeatures_mask_all = cpu_user_xfeatures_mask | + cpu_system_xfeatures_mask; + + if ((xfeatures_mask_all & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -760,10 +777,11 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask_user &= ~BIT_ULL(i); + xfeatures_mask_all &= ~BIT_ULL(i); } - xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_all &= SUPPORTED_XFEATURES_MASK; + xfeatures_mask_user = xfeatures_mask_all & cpu_user_xfeatures_mask; /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -775,8 +793,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, - xfeatures_mask_user & ~XFEATURE_MASK_SYSTEM); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -784,7 +801,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask_user, + xfeatures_mask_all, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -804,6 +821,13 @@ void fpu__resume_cpu(void) */ if (boot_cpu_has(X86_FEATURE_XSAVE)) xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * Restore IA32_XSS + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, + xfeatures_mask_all & ~xfeatures_mask_user); } /* @@ -853,9 +877,9 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) /* * We should not ever be requesting features that we * have not enabled. Remember that pcntxt_mask is - * what we write to the XCR0 register. + * what we write to the XCR0 | IA32_XSS registers. */ - WARN_ONCE(!(xfeatures_mask_user & xstate_feature), + WARN_ONCE(!(xfeatures_mask_all & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -1005,7 +1029,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SYSTEM; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1089,7 +1113,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SYSTEM; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1182,7 +1206,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: @@ -1238,7 +1262,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SYSTEM; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: From patchwork Fri Sep 21 15:03:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610221 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 202EB15A6 for ; Fri, 21 Sep 2018 15:08:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C7E62E421 for ; Fri, 21 Sep 2018 15:08:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0045F2E423; Fri, 21 Sep 2018 15:08:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 66C512E421 for ; Fri, 21 Sep 2018 15:08:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AB828E0008; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 25AF18E0001; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 060518E0008; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id B5D8B8E0001 for ; Fri, 21 Sep 2018 11:08:50 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id o27-v6so6714611pfj.6 for ; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=HYv72t+ibUOa7b6wsMKk0mmqvwO5mzmA3sLIsJC1ojE=; b=gTshnVZLO7uCwLnL8K998xeyRg7ahfFQ3l7gVsrau5qQIZ8rWv6H/qXhpzizkNx5pi amn/L3GLE1kVBxrC8mGSRxEze6H+E0AOqgRKedgwMTzAMp/RhSOLTV4HYf03+UYYXLLq X5iqN0RUrDXCHzqxOVAa+BvYlBOo9WlvABae9QPJmmrqCbMr8+VzyCMaM9mryYh3tvzw kUBG9wlMqlUNOpO/DBhGbzp9LJ5qSIKL1sK9l8sb+DnuQOTqYheuVEC41ggFjRr61T3I KGjRPvPnpyYO69NT51OFPBqVpWvWJcv0bWpWl2LW8NmE1P0hSeY/DL1m2u23KahxXg/K lpsA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CRp3u+PquMN9GsB8RVw2TNnQt31j7kYlhmko7/FbtBVyHSpXjS crDr05C4gUHOrI7V8FFB3zAi3fljvg4SzCc78JE7Zs+kdVgwaBKkFJ0E6ED+Qrn8t3LPtdbu6rN eFuiJi9YmX67JnDhft68HqdELKYffwSP8w8menhkB05e6FV9akBCOYppoVU3dnSKksQ== X-Received: by 2002:a62:3001:: with SMTP id w1-v6mr46849528pfw.19.1537542530418; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda/XVWCSNYSJwdZ4Oo8v1GGNLY66U6SP7lnBCyUhttd0B5uK/ciHxFEdMa/3TyFEB1krfdP X-Received: by 2002:a62:3001:: with SMTP id w1-v6mr46849463pfw.19.1537542529297; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=s2vmwJ/kLWwAXzEXtaVk3j/zIskOX0glZ1w8Cr8EHASk/TGkhNuRvsgXsDmY8fkmhF fWT6kVRXGIT3QA468LhEuLjCPhvMig221cSLL1v/RrXCONmS5rZsAnPfvZ+LaIvm7F9S boXI49h5WfB04YJNwT1IN/uuLBCa5RKgjZT4DeaD0K5Zi/pwjadH2fB8OtFPG4flLwe+ pvQw+aeM4baIoI9C7bWb+fWqvRzTfUvUCMhMS4W54ggtBqM8rUpTyUQ6z5Dzh420m3jl ewiWqP7jXSTHyJeBnzdqqBMFkFl77iFp2so02E5IyqsIF/l0Jed/pV+tUQSCgh/AuzQh 1K2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=HYv72t+ibUOa7b6wsMKk0mmqvwO5mzmA3sLIsJC1ojE=; b=L88JzRdm1JEDGwpHb6MeBOv/5vXWfeZzyDwROwy3/qGBmNXJ1I40PnvH4vIf10TJ9i DSP5zxq4eQVmLFIzzQbOeZTAMHD8+vJvpqB4JaNAHZzT91598lJH2YRckzeh9r5FQGq7 bIaVku2rkAEra0JL684ij/hkMK+iXs6Se3nJ5n7z5ex39+cpxVcEThLXZiZJLkAW9pLu qtm7VVYoEpH9UrGjvu1ADBEmPZjSlox5NakNkFNRn+HPsn0JRG1tio46HAItwVsgfuAO SOLNBxLbQAYKJl6B4LNi1hRx6UFVvJpxeniWgADQ9lZjxPPQKiaZYixFaiOwwaigE6It IVnA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id g2-v6si27080525plq.242.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856539" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 04/27] x86/fpu/xstate: Add XSAVES system states for shadow stack Date: Fri, 21 Sep 2018 08:03:28 -0700 Message-Id: <20180921150351.20898-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Intel Control-flow Enforcement Technology (CET) introduces the following MSRs into the XSAVES system states. IA32_U_CET (user-mode CET settings), IA32_PL3_SSP (user-mode shadow stack), IA32_PL0_SSP (kernel-mode shadow stack), IA32_PL1_SSP (ring-1 shadow stack), IA32_PL2_SSP (ring-2 shadow stack). Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index 202c53918ecf..e55d51d172f1 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM, XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, + XFEATURE_RESERVED, + XFEATURE_SHSTK_USER, + XFEATURE_SHSTK_KERNEL, XFEATURE_MAX, }; @@ -128,6 +131,8 @@ enum xfeature { #define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM) #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_SHSTK_USER (1 << XFEATURE_SHSTK_USER) +#define XFEATURE_MASK_SHSTK_KERNEL (1 << XFEATURE_SHSTK_KERNEL) #define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE) #define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \ @@ -229,6 +234,23 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control flow Enforcement user states + */ +struct cet_user_state { + u64 u_cet; /* user control flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +} __packed; + +/* + * State component 12 is Control flow Enforcement kernel states + */ +struct cet_kernel_state { + u64 kernel_ssp; /* kernel shadow stack */ + u64 pl1_ssp; /* ring-1 shadow stack */ + u64 pl2_ssp; /* ring-2 shadow stack */ +} __packed; + struct xstate_header { u64 xfeatures; u64 xcomp_bv; diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index a32dc5f8c963..662562cbafe9 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -31,7 +31,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_SHSTK_USER | \ + XFEATURE_MASK_SHSTK_KERNEL) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..25311ec4b731 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -130,6 +130,8 @@ #define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT) #define X86_CR4_PKE_BIT 22 /* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23 /* enable Control flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) /* * x86-64 Task Priority Register, CR8 diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index dd2c561c4544..91c0f665567b 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -35,6 +35,9 @@ static const char *xfeature_names[] = "Processor Trace (unused)" , "Protection Keys User registers", "unknown xstate feature" , + "Control flow User registers" , + "Control flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = { @@ -48,6 +51,9 @@ static short xsave_cpuid_features[] __initdata = { X86_FEATURE_AVX512F, X86_FEATURE_INTEL_PT, X86_FEATURE_PKU, + 0, /* Unused */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_KERNEL */ }; /* @@ -316,6 +322,8 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_ZMM_Hi256); print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); + print_xstate_feature(XFEATURE_MASK_SHSTK_USER); + print_xstate_feature(XFEATURE_MASK_SHSTK_KERNEL); } /* @@ -562,6 +570,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_KERNEL, struct cet_kernel_state); /* * Make *SURE* to add any feature numbers in below if From patchwork Fri Sep 21 15:03:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610245 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 483045A4 for ; Fri, 21 Sep 2018 15:09:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 357542E429 for ; Fri, 21 Sep 2018 15:09:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 294852E42E; Fri, 21 Sep 2018 15:09:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BA0F2E429 for ; Fri, 21 Sep 2018 15:09:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 658CB8E0017; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4B9308E000C; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C245E8E0014; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id A5BBD8E000F for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id b29-v6so6735636pfm.1 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=zTtarEDYJFAJ15G4EkXt7q3hAoZWqXZZ+JIBsb4Teco=; b=n+0hkLzB0sq+kM6Z27dWIaW9OJXdj2P2cEIAW7pC912XKw6/J+pTS+3A7eszpx5dbS o9avu3F5YpMTrecGxFpUycWGYsTKbceSE67ntB6dOfnmbwlMVBUrIwEGpx+kZKSIDDHT tCFuzAtlNNI/1BIw3DILYZ7ntTukGSLpkgFT97bvLcqLqcshELRr6yWnQ6pTztiy6KTr tRKF3ZzzASEe0al22Zn0ujrkV0in+0Pkyba++g9vclrPMWeHR5VqLmqp74h+6vVoCvPa kVdoU1YjpzoeoP6OeQQDQGzbLNkxrMCT7856u+kTazMJJMdlnBmhR0KxibIq+5ZjJwIj lzqQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BdZTToRYcSAGfh+4A+EBCcSataACUQ5Yf+IQMGkxnlfVrifjQz D0vIxnxZEet93Xdu0RqRiUNJ4rGkMgGHKDdcCfuw+4GxI6QU5GLhyGa6Kjo0z08prMsvfTemrHb w4on3hNm3BOMrsPEfPWOeBL67rNAPDKJ59b42/0LxSIpJyjk/4OMQ1gm5zLzfl9knIw== X-Received: by 2002:a62:d94:: with SMTP id 20-v6mr46733852pfn.202.1537542531319; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaU3CYQcv+7AAwdktmZkYxemPJlNeNlHm8gxlI999syjuXVaEkhF9Y5w7rUMcQelwgAGsCn X-Received: by 2002:a62:d94:: with SMTP id 20-v6mr46733734pfn.202.1537542529551; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=qok3SBJHtC5Yh5yH6BBD2hiEXmsWBTa98mOJxhUgw8d389t4gZu2B1la+Y/bTkWvey 4PoK92a/JwTXkB6fFTVaEiC2HRLc8519EwyYwj0RsDNFB43ohOYGFTPfqrJWokN/vs7r dFL8zygjQyznFKoeEJIR0zvDUTiokSH7x6kwykISSaJFc55FRtTEq/nieSDFVe7lk1uA v/rgnwq/e5wz84+RxzvERqJYlz/JiDnG1UVFqK5S2evyulIzOpJCBtuM+/YpGAPs20RE VPkwdH72rrK1Ah4KzJ+p8k1tJt2t9MOB7DhCwJpjAclNfr+i4LtfZjF8t730RrgwHgL0 Jo1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=zTtarEDYJFAJ15G4EkXt7q3hAoZWqXZZ+JIBsb4Teco=; b=yGtYndm2NhzeI0/KEbBxgE2HP76vXc8MuF27zquIx9zbS9KxSLrKViR1SvcssQi24g OlfudNZU81gXBI53G12iuwJRurJUt4t6aYK6w60DcnK/yvQsb7floTYvQAEJVVrog0sk zAr7/IPxvuaJ2NDccxXkLudnEUNlFPYbY6YNXEp+vXWwS+DmNreR/1zSNRkRfCXkzbCu jE5vnnJifRajIwUKbc+tF8DZ2S1jRDLg5EI+i/SQ4mLt3vhWivVe5L004PdujcIAK2Dj bKJIU1CiUmLLq/oRWBlSq6rw8a/+GKYs/NSx+538Stc5uXXkeLtMOUMnkf5qG0kqRejo doGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856543" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:47 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 05/27] Documentation/x86: Add CET description Date: Fri, 21 Sep 2018 08:03:29 -0700 Message-Id: <20180921150351.20898-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Explain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters. Signed-off-by: Yu-cheng Yu --- .../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 11 + Documentation/x86/intel_cet.rst | 259 ++++++++++++++++++ 4 files changed, 277 insertions(+) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 92eb1f42240d..3854423f7c86 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. diff --git a/Documentation/index.rst b/Documentation/index.rst index 5db7e87c7cb1..1cdc139adb40 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -104,6 +104,7 @@ implementation. :maxdepth: 2 sh/index + x86/index Filesystem Documentation ------------------------ diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst new file mode 100644 index 000000000000..9c34d8cbc8f0 --- /dev/null +++ b/Documentation/x86/index.rst @@ -0,0 +1,11 @@ +======================= +X86 Documentation +======================= + +Control Flow Enforcement +======================== + +.. toctree:: + :maxdepth: 1 + + intel_cet diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst new file mode 100644 index 000000000000..56e724fce920 --- /dev/null +++ b/Documentation/x86/intel_cet.rst @@ -0,0 +1,259 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protection against +return/jump-oriented programming (ROP) attacks. It can be implemented +to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented on the 64-bit kernel. +However, 32-bit applications are supported under the compatibility +mode. + +CET includes shadow stack (SHSTK) and indirect branch tracking (IBT). +The SHSTK is a secondary stack allocated from memory. The processor +automatically pushes/pops a secure copy to the SHSTK every return +address and, by comparing the secure copy to the program stack copy, +verifies function returns are as intended. The IBT verifies all +indirect CALL/JMP targets are intended and marked by the compiler with +'ENDBR' op codes. + +There are two kernel configuration options: + + INTEL_X86_SHADOW_STACK_USER, and + INTEL_X86_BRANCH_TRACKING_USER. + +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later +are required. To build a CET-enabled application, GLIBC v2.28 or +later is also required. + +There are two command-line options for disabling CET features: + + no_cet_shstk - disables SHSTK, and + no_cet_ibt - disables IBT. + +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT. + +[2] CET assembly instructions +============================= + +RDSSP %r + Read the SHSTK pointer into %r. + +INCSSP %r + Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated + in the operand register. The GLIBC longjmp uses INCSSP to unwind + the SHSTK until that matches the program stack. When it is + necessary to unwind beyond 255 steps, longjmp divides and repeats + the process. + +RSTORSSP (%r) + Switch to the SHSTK indicated in the 'restore token' pointed by + the operand register and replace the 'restore token' with a new + token to be saved (with SAVEPREVSSP) for the outgoing SHSTK. + +:: + + Before RSTORSSP + + Incoming SHSTK Current/Outgoing SHSTK + + |----------------------| |----------------------| + addr=x | | ssp-> | | + |----------------------| |----------------------| + (%r)-> | rstor_token=(x|Lg) | addr=y-8 | | + |----------------------| |----------------------| + + After RSTORSSP + + |----------------------| |----------------------| + | | | | + |----------------------| |----------------------| + ssp-> | rstor_token=(y|Bz|Lg)| addr=y-8 | | + |----------------------| |----------------------| + + note: + 1. Only valid addresses and restore tokens can be on the + user-mode SHSTK. + 2. A token is always of type u64 and must align to u64. + 3. The incoming SHSTK pointer in a rstor_token must point to + immediately above the token. + 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK. + 5. 'Bz' is bit[1] of a rstor_token indicating the token is to + be used only for the next SAVEPREVSSP and invalid for the + RSTORSSP. + +SAVEPREVSSP + Store the SHSTK 'restore token' pointed by + (current_SHSTK_pointer + 8). + +:: + + After SAVEPREVSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) | + |----------------------| |----------------------| + +WRUSS %r0, (%r1) + Write the value in %r0 to the SHSTK address pointed by (%r1). + This is a kernel-mode only instruction. + +ENDBR + The compiler inserts an ENDBR at all valid branch targets. Any + CALL/JMP to a target without an ENDBR triggers a control + protection fault. + +[3] Application Enabling +======================== + +An application's CET capability is marked in its ELF header and can +be verified from the following command output, in the +NT_GNU_PROPERTY_TYPE_0 field: + + readelf -n + +If an application supports CET and is statically linked, it will run +with CET protection. If the application needs any shared libraries, +the loader checks all dependencies and enables CET only when all +requirements are met. + +[4] Legacy Libraries +==================== + +GLIBC provides a few tunables for backward compatibility. + +GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT + Turn off SHSTK/IBT for the current shell. + +GLIBC_TUNABLES=glibc.tune.x86_shstk= + This controls how dlopen() handles SHSTK legacy libraries: + on: continue with SHSTK enabled; + permissive: continue with SHSTK off. + +[5] CET system calls +==================== + +The following arch_prctl() system calls are added for CET: + +arch_prctl(ARCH_CET_STATUS, unsigned long *addr) + Return CET feature status. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = SHSTK/IBT status + *(addr + 1) = SHSTK base address + *(addr + 2) = SHSTK size + +arch_prctl(ARCH_CET_DISABLE, unsigned long features) + Disable SHSTK and/or IBT specified in 'features'. Return -EPERM + if CET is locked. + +arch_prctl(ARCH_CET_LOCK) + Lock in CET feature. + +arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr) + Allocate a new SHSTK and put a restore token at top. + + The parameter 'addr' is a pointer to a user buffer and indicates + the desired SHSTK size to allocate. On returning to the caller, + the kernel fills *addr with the base address of the new SHSTK. + +arch_prctl(ARCH_CET_LEGACY_BITMAP, unsigned long *addr) + Allocate an IBT legacy code bitmap if the current task does not + have one. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = IBT bitmap base address + *(addr + 1) = IBT bitmap size + +[6] The implementation of the SHSTK +=================================== + +SHSTK size +---------- + +A task's SHSTK is allocated from memory to a fixed size of +RLIMIT_STACK. + +Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, we can use a large SHSTK to +cover the condition that both the program stack and the sigaltstack +run out. + +The kernel creates a restore token at the SHSTK restoring address and +verifies that token when restoring from the signal handler. + +Fork +---- + +The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be +read-only and dirty. When a SHSTK PTE is not present, RO, and dirty, +a SHSTK access triggers a page fault with an additional SHSTK bit set +in the page fault error code. + +When a task forks a child, its SHSTK PTEs are copied and both the +parent's and the child's SHSTK PTEs are cleared of the dirty bit. +Upon the next SHSTK access, the resulting SHSTK page fault is handled +by page copy/re-use. + +When a pthread child is created, the kernel allocates a new SHSTK for +the new thread. + +Setjmp/Longjmp +-------------- + +Longjmp unwinds SHSTK until it matches the program stack. + +Ucontext +-------- + +In GLIBC, getcontext/setcontext is implemented in similar way as +setjmp/longjmp. + +When makecontext creates a new ucontext, a new SHSTK is allocated for +that context with ARCH_CET_ALLOC_SHSTK the syscall. The kernel +creates a restore token at the top of the new SHSTK and the user-mode +code switches to the new SHSTK with the RSTORSSP instruction. + +[7] The management of read-only & dirty PTEs for SHSTK +====================================================== + +A RO and dirty PTE exists in the following cases: + +(a) A page is modified and then shared with a fork()'ed child; +(b) A R/O page that has been COW'ed; +(c) A SHSTK page. + +The processor only checks the dirty bit for (c). To prevent the use +of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as +DIRTY_SW for (a) and (b) above. This results to the following PTE +settings: + +Modified PTE: (R/W + DIRTY_HW) +Modified and shared PTE: (R/O + DIRTY_SW) +R/O PTE, COW'ed: (R/O + DIRTY_SW) +SHSTK PTE: (R/O + DIRTY_HW) +SHSTK PTE, COW'ed: (R/O + DIRTY_HW) +SHSTK PTE, shared: (R/O + DIRTY_SW) + +Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs. + +[8] The implementation of IBT +============================= + +The kernel provides IBT support in mmap() of the legacy code bit map. +However, the management of the bitmap is done in the GLIBC or the +application. From patchwork Fri Sep 21 15:03:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610225 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB9075A4 for ; Fri, 21 Sep 2018 15:09:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D98652E421 for ; Fri, 21 Sep 2018 15:09:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC7AD2E423; Fri, 21 Sep 2018 15:09:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36A872E421 for ; Fri, 21 Sep 2018 15:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E868F8E0013; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D57278E000B; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 873D88E000A; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 348D58E000C for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id w18-v6so6317156plp.3 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=9q6n5bxbKSrI+N/Uw4m6pM8HmTZU567xn34Kn3sTlFc=; b=DwuyBC2rPdu/gL6I4ilfJ3VQKwZWYHd0tV2K/IxrVCvkDW+FAWjiO0c//5cv5xazyp HLN+Hoavbp7/7b8tUmIyFuYOKRGTs5NU0ND1Fsmq/RbHJlFnyohCMFwYGrDhX+vUgByH HFCGNA7AG8Sug6CnZdNIb3nPxT1E6VUaCP9jBXR/m1jIB3SA+wjzF5Zl4GEP2RK4HodP xBRv/TbUYTawyoVdo4hy4pT8aWn04GjyKxEL0ZcnjXW4fiPDvmvrB8qMWfoPy1N95QvV k9irTaYK4zsbB3aERYH39wm1iXTqYE+q3FuKs5FPMyHXPi3pbTteKKRTitCFTb6hhTj8 3SSg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51B6erFtlbhQ3IDVhp6YPRXRHA8VXHAyf8QN4QlrUxpkQFa5n40q VsSEtVYp1Ek2O0vVvfqQQsewk4TQh0ml85XOrpxUQdLNDcvB53Sjr9mt2sdP7IQ2tm2QXaT/R0K cKjeWyzJ9DStO7pMb07iH1c00QvcQxHRoCGzwssKs2nNWRSGjqvv9Jhsw6yVQITfHYA== X-Received: by 2002:a63:b705:: with SMTP id t5-v6mr39553241pgf.366.1537542530785; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdaxq+m7ZJmUfmorlfZdfkq2ArWj77Wv8a/y3wu9nZNM8+8vHfT6St2Vr8B/hJnN2nAD/WuA X-Received: by 2002:a63:b705:: with SMTP id t5-v6mr39553166pgf.366.1537542529577; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=Jzn3Utz77Nr4XneBMLPVW5THRqobXFQFMuWEC2H/OHHD5XMi1YWaUEMCYF/MhVWCg/ tzE58w0UMLJmv3nn9FGb9kWj46kgpxb7SkwHlBe96w0RUGdWRVmCus7QiU1fUSGB+1o/ DJnguP1KJG22yAGOuf5ogluJb/vEDk49UY+Ne9L7yzStcWh46hMbKm7Kvn0PfzHX0I8P 5bTcXR/ewA4Rkke+VVqmfeUiio5V/AqWCieELeZD2E3HNbmwgs90lvzFCGgp0P0td5Kt 8V0dsEUgYo1jE/xKGRQrN/yeqtNe/S79m3FTGgFrCx9hjC+eEumMcTPWTqSIbmBfFR/D Nwug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=9q6n5bxbKSrI+N/Uw4m6pM8HmTZU567xn34Kn3sTlFc=; b=FOHmCrU7KaOFqaeuT7yWc+lmbxFsY54GJKBxTbT/231lDRurQtRDXg7XhL+wMsdbwe JTfv3f49vlh49ALmg5wRKIwxoLTFSbnJF709FK0xm4xnRxZuUfuPh1ri21ce0T6IC4pB zAglrgmYkJxKNzMcN+ytcQ/hYFDjlTuEGHIRc+02Zz2w1UW49w25dj3xSVC6/JL3Q5XX rZEHHVryt+QP0c9o9qW3u3W3WPLE1aD9pSJOpfGAQWACYBPLpqf0ITdHeQ9z7LIybrod ihWun8BC/j1rHgAcnTwjvsqm8SjWQMmASvY/wK+5mQDWb754rCD4GYFoDb5MinBYcGz/ pBOw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id g2-v6si27080525plq.242.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856546" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 06/27] x86/cet: Control protection exception handler Date: Fri, 21 Sep 2018 08:03:30 -0700 Message-Id: <20180921150351.20898-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction. The control protection exception handler works in a similar way as the general protection fault handler. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/traps.c | 58 ++++++++++++++++++++++++++++++++++++ 4 files changed, 66 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 957dfb693ecc..5f4914e988df 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -1000,7 +1000,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 3de69330e6c5..5196050ff3d5 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void); @@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long); #endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *, long); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long); @@ -142,6 +144,7 @@ enum { X86_TRAP_AC, /* 17, Alignment Check */ X86_TRAP_MC, /* 18, Machine Check */ X86_TRAP_XF, /* 19, SIMD Floating-Point Exception */ + X86_TRAP_CP = 21, /* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ }; diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 01adea278a71..66ebc8cb16e2 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = { #elif defined(CONFIG_X86_32) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32), #endif + +#ifdef CONFIG_X86_64 + INTG(X86_TRAP_CP, control_protection), +#endif }; /* diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index e6db475164ed..873765adc244 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -578,6 +578,64 @@ do_general_protection(struct pt_regs *regs, long error_code) } NOKPROBE_SYMBOL(do_general_protection); +static const char *control_protection_err[] = +{ + "unknown", + "near-ret", + "far-ret/iret", + "endbranch", + "rstorssp", + "setssbsy", +}; + +/* + * When a control protection exception occurs, send a signal + * to the responsible application. Currently, control + * protection is only enabled for the user mode. This + * exception should not come from the kernel mode. + */ +dotraplinkage void +do_control_protection(struct pt_regs *regs, long error_code) +{ + struct task_struct *tsk; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); + if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code, X86_TRAP_CP, SIGSEGV) == NOTIFY_STOP) + return; + cond_local_irq_enable(regs); + + if (!user_mode(regs)) + die("kernel control protection fault", regs, error_code); + + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control " + "protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + printk_ratelimit()) { + unsigned int max_err; + + max_err = ARRAY_SIZE(control_protection_err) - 1; + if ((error_code < 0) || (error_code > max_err)) + error_code = 0; + pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); + } + + force_sig_info(SIGSEGV, SEND_SIG_PRIV, tsk); +} +NOKPROBE_SYMBOL(do_control_protection); + dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) { #ifdef CONFIG_DYNAMIC_FTRACE From patchwork Fri Sep 21 15:03:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610223 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C45EC15A6 for ; Fri, 21 Sep 2018 15:08:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 622982E421 for ; Fri, 21 Sep 2018 15:08:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55B822E423; Fri, 21 Sep 2018 15:08:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D39A32E421 for ; Fri, 21 Sep 2018 15:08:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6A878E0011; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B3668E0010; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B4938E000F; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 2E55B8E000A for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id o27-v6so6714618pfj.6 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=SrAoz3B3fVnohd0Wm1rBW1k5UQJ9ZmOZzebnbQuoFhw=; b=idqY6PsIOGJXnPWLoGN9MQ+MFGuA+8+3qWk3eHYsr4VHRi3lQ60PvchEPPUz/cMN/b DP3qi8cf0L7wmj+4rUeXvNJlFJqDgZfKdy+3Jtj/UqvTzJNsAVA3SRnt1r1wSnL12dYJ wbypJzz89Vczc9kJx8hE01DLXUu8iwCHkFXn50+0y/fQ7C2VM0B+gGP2o2N+yikDVHlk 5TjoNTQUNqn4sC5HE6oH+LLwwtarKfNRhvdDzZSa76nn1iFLLJ4o0W77oCDLzGT8LT4K qxv4XWbEZ8qZ/JbxGYzzcQ0ktipvJ9ji+ioG3Kx8PZyMHhDT4i7+93zEY7DSE/Z6GfqK /iGQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BY5BD5gSmDoBbTVlO5o1/K5Tpg3+5W9SSMhu1++HDNFcJj+gTI njZ7Sz2MS8IGrpgGW1FTK+ch5tHHaGiMzkqU8khWLrk9BuTIEHuasGSwYNueOP+rNb4aVK7rlEG p181dydBZZY+ySHqhOSF1BrE7R6SAVyUv7rn8THSZPhYsBsbyHXDeSUqOjeo+Jpx/Rw== X-Received: by 2002:a62:45d2:: with SMTP id n79-v6mr46933593pfi.137.1537542530813; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaMnD/Vnlk02IPv2B+UG0t7G9V0tUWw3q92nzCkAFIZSognSICZdXhaPI+bN2s3yEMyq0j7 X-Received: by 2002:a62:45d2:: with SMTP id n79-v6mr46933523pfi.137.1537542529808; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=A0DqJQpjqSuY+ri5OX6gtfzqDJuQn79M6m80SwNrr8BfRoGsmpuV/8rSBxia+XWFE8 12fs0bFCeNZQXLkNO0IdnDDrFJAfPCCNN9rNV//vo1uTGhzAkWFVTMECiiaFS1A/J3dr exRAWtDjt/wPzMfofmk8Edo5elJ564NgE5vna4Yq7nwhWBNws2YUvbNy5iAUUSypVlG/ lLudQseqX2xbTYFvBe7D/Kv+K3t41NYK54+5QKHQsl0ph2i7TdmC3bJ9CTOqB6b/Gzq0 pM/3q1YR6aW+ka4BuGjs8/4P2RgE1DRjejsNUIBOen+2sXJjmEMyfpxBA7G4yV0vAfbq a+4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=SrAoz3B3fVnohd0Wm1rBW1k5UQJ9ZmOZzebnbQuoFhw=; b=nnRrKyTeu07VTO8E5kcjWPI3sody2PAofIuL+z/9LF+nkrCg6hcNBLp/ILnHxoz1qU 64PSG4hkYDDspJ2BS0/waLYB6RqKx2vAP8RDhi7yIvopFKTom7Jb484mfI/hWX19vB1H NsEaq1wbpwyc1EQzMO+P0+uwyC0gmHiRNRcMXVn/B3iQfIC8NUz0EDlRua6C2LV7/fHq ieCeErhJdU94XHfZtmaT1RZeVCmzZ6BFMZ38bTEIoA5/j/oyVUtsUM7cWpfjC7HT8+v0 uJphzrY0S2+aaco+VUf/vlWzkvGISDZW2MHPTscCMymI3kZWDxIkPBPGDZjq1g7nVjkL tCIw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856549" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 07/27] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Date: Fri, 21 Sep 2018 08:03:31 -0700 Message-Id: <20180921150351.20898-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER. An application has shadow stack protection when all the following are true: (1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled, (2) The running processor supports the shadow stack, (3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack. If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection. The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 31 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..808aa3aecf3c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1913,6 +1913,30 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select X86_INTEL_CET + select ARCH_HAS_SHSTK + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 8f6e7eb8ae9f..b28842b80295 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -152,6 +152,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Fri Sep 21 15:03:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610227 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 560D25A4 for ; Fri, 21 Sep 2018 15:09:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45DE22E421 for ; Fri, 21 Sep 2018 15:09:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39EEC2E423; Fri, 21 Sep 2018 15:09:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C9FE2E421 for ; Fri, 21 Sep 2018 15:09:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EAF48E000D; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2AA618E000B; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A592E8E000D; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 408C88E000D for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id r130-v6so5803911pgr.13 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=g+8VbSksYbQBAeUV/pGpbdGejDXNcTltJ5zVBXaYRSU=; b=a54Nbnfc/PxXEQCQx9okLrHdPIJnGMs4VUOJzvL7AnNmXF8e3/5suPiJnpNTbXuD/M DhZMfBMhLZ2kISDPSLBcXvbAxdwXJigYrZyxljAXDF4mQR5sC2wWujMOjVHA42/qHZIi 0M4Qs3kRn+VUFV1HJwHYhfwpeAR44z8xHrUDZEHYb0rBMEDESd8C8aJfSCsq051Fwqjk wPyP8XvHctarJr9vviPS7twXJMpRXynoDc5KTdxpgyzGJB1RU/RyHr+hHkst6QAVDubC rGVly/J4T2QGh/GlP7KXx6iRkrfabsuXnql55z22Crs/q2FnhwBGAciRNJd4hYLmlOyP xPdw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CupwJTVChyOLO8kETsEwTGabb1eIkYxE9sD8T3SqxSFQtQDxcg 5zDPK8coP5EY7udVMZWzVt6D72k0JXnWH/XlBpXxcUQadk/82UrE5ZWnQdG+eG0LSr7VmkkTNbX TFNZV179S6I8lfxqB9fZEQ6Txz7XUzpPb1/+GckHXmwl61sJbwGpoRKNaiZL8o7iM6Q== X-Received: by 2002:a17:902:bf46:: with SMTP id u6-v6mr4582095pls.85.1537542530888; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) X-Google-Smtp-Source: ANB0VdakpsI7WIJEAR0YuY7GJZqUDlYGN4FEPvhU3fGhJiIaDVNk86D2RwShhgzbyqvCHMfhPhWN X-Received: by 2002:a17:902:bf46:: with SMTP id u6-v6mr4582042pls.85.1537542529914; Fri, 21 Sep 2018 08:08:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542529; cv=none; d=google.com; s=arc-20160816; b=Q0MaTG0vs3zA3UC0TV0QWhfk+NmiaKivkPP5l/Sr74A/fY2pYHCvHdDe4ysc+DmX0X JcoD93M/MdewN5Z9xbuKDLuoi9V0ljhDz8PTIu3ffMmS0FTqQ4kGjkWsa9z4THICYVoq tVVfS8ccbUbfaBCxWopt3exAwqUgc4Z/HP3FrMH5DU8SxItuI832nqZdA4vPK8ehAnv1 qjJJ+XxdH6nSTa+qbwL36r74sMOo5a9NSYKxJrjN9HZKdPR9Vows56hbABaQ72gFdQBd GNwLN36HhdWXRb1zvqTDwEO6CmUIfUo9SsFT3poJ83QTxY1b0zNxHZYKsle4x6NCpkL6 Hc0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=g+8VbSksYbQBAeUV/pGpbdGejDXNcTltJ5zVBXaYRSU=; b=KxwL7YMpDtcshrcYNdJ0udwByianFPjsDvp4QVc5/ARMAo0fBcp158F/HtyP31BY7h tnemXNMbAw9vR3jGBv9r/TwI51/sTx97lr58daEMCjsQ1Xa+sH6ILfbykeC1iG97EoIO UuFKLYHi10chR4ARNorxZRNT2p1mGcW8GmnFSdyIMFrS6kchaY25dhNvsrfgUvi+w4BX uRwmYCzTttq6uKov5imYTuxALygFknq8247SsLhaQ/ll4FWt6le2jpqSC8FHZSvnsAre FyCpYIyBReog6x/FvYny+068zv7nfgS5wefC7kmeM4Fss33H8UhU6SlBRGOLQXpUm2mf +YmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id g2-v6si27080525plq.242.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:49 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856552" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 08/27] mm: Introduce VM_SHSTK for shadow stack memory Date: Fri, 21 Sep 2018 08:03:32 -0700 Message-Id: <20180921150351.20898-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP VM_SHSTK indicates a shadow stack memory area. The shadow stack is implemented only for the 64-bit kernel. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..f40387ecd920 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -224,11 +224,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -266,6 +268,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_MPX VM_NONE #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif From patchwork Fri Sep 21 15:03:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610243 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C24AB5A4 for ; Fri, 21 Sep 2018 15:09:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B08AB2E429 for ; Fri, 21 Sep 2018 15:09:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A45352E42E; Fri, 21 Sep 2018 15:09:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8302E2E429 for ; Fri, 21 Sep 2018 15:09:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2495C8E000F; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E8F658E0015; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E6AD8E000A; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 8D96C8E000C for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id 191-v6so5737035pgb.23 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Tj5+cMyMtxeZvdRe/q9Y3Ze47G1cE17sIqrN+h5d9Xg=; b=HsuYGEgW+W2PfONg0MfMI6nQiJGOg4fAMTbt+D2InfBH8mZFPk84DQVUOcKDbmOusH 1rqT5YpeoUJJjQRE4ZNA33DggCqqx1bUijQXOCGVyvi6nhIcLdEnhUjYFgIsv4gX3DiQ UAi6eYrTOIK2DWVgvtG1awUkiLcg/RKR4e4P1wDpnSBeQzOwX+fEDXq20cGuCP4puJVg keOM7mJOX6MhpR7lUxOn4EoBPwzGC6cShqaRUWxtcbqhW/61+NiUb8afecwNLLfLUD40 wcv1ODHEVM8HP3T0B8EFbofeNqvfsjMR8bpF7SFQjwi0RwOOCpGVAZO/gKa1ms2RQBhs 9tEA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DzSRzoeabnARMyhKgsAhxCV6sfvE7PF0IPDaT0tNBPK7bHXXVB 0hTWwSSznhK4MVGN57ck127plfG9ZuhqkvUEPKDjB5RUDVNuhDAjcG/HdrbS/8gJdc4JQChTQq6 kA79dOhAFo7TJ8R1BMys8fJRP1lCEh9sGh55Ro2AP2hbVzcn3gjkiPdyHaCE0N3hJdw== X-Received: by 2002:a17:902:e85:: with SMTP id 5-v6mr43707192plx.73.1537542531245; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdabmWHAd1ani3hdE5QixE2eBEb9+bQ4hG9QVdPtA40zhrIqwhlEAvmROXI00YP/nwXQMohl X-Received: by 2002:a17:902:e85:: with SMTP id 5-v6mr43707135plx.73.1537542530085; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542530; cv=none; d=google.com; s=arc-20160816; b=RpPUwzkezXg6Iwy2RbVcsPkweOl31gAnSxbtDiTaj/9DwvTRJpoIQEefSLIYbu4j8/ m2jBa6eus8CX0woxGQANHqs7e+ThEMfkjuhmzpW5gjfB+xcvL0EPdaJ6ZBd9OJtbLgY2 UpRXua/eG7sIsb17oRWUXuTflBwcf8CnPTnza8zrv3oAXNS6UbHOGRKtNK7lVZcz35mg 75JRk2VFG+yCI94DsuaK7nYJkcDzLCmmO508PF8lQ56ThLjDp+bH6PHQWiPParEQThPe AjV/6XeZWWL/B0Fz8dX4Xc6ZpQge4WCxXLwQUMvTKKNLLMGfwAr0iOP1ZDGubs8ZP6zS H6QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Tj5+cMyMtxeZvdRe/q9Y3Ze47G1cE17sIqrN+h5d9Xg=; b=q0iTzh4XhbgePd+nrOYLAOqxA5xR6obXXYpmpM1HgFpwVzSukgrMdZrGezyfGrVhal TI74qFpLrGzc5VoaharcJeTCf73PjEb2igFW+8e7Wz2F+cLKe2I8HoL3XnKwW7rHjBBi DCPNOWtHr/8o2BcvOEQky3ZRsWbW4PF7oA5BUiZGG9oZ9bvcQE/eYdSP4N3SaxcB/86z +Lli8MPsMmehcqMwTuvY9EdMRVd/QeVd00tGyir6ZFvw3Rt1naBFs11qxVQW1zFfDZD/ oOkEZngmoTmqRuR2YReuIaoS15fDZABf6H8OgpgPd8kYG9HqpqD8LQhBdn++je6yXapk d5kg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:50 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856555" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 09/27] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Date: Fri, 21 Sep 2018 08:03:33 -0700 Message-Id: <20180921150351.20898-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We are going to create _PAGE_DIRTY_SW for non-hardware, memory management purposes. Rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 690c0307afed..95c918ad84ed 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -316,7 +316,7 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) @@ -390,7 +390,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -444,7 +444,7 @@ static inline pud_t pud_wrprotect(pud_t pud) static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index b64acb08a62b..0657a22d5216 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -15,7 +15,7 @@ #define _PAGE_BIT_PWT 3 /* page write through */ #define _PAGE_BIT_PCD 4 /* page cache disabled */ #define _PAGE_BIT_ACCESSED 5 /* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6 /* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6 /* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7 /* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7 /* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8 /* Global TLB entry PPro+ */ @@ -45,7 +45,7 @@ #define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT) #define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD) #define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW) #define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE) #define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL) #define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1) @@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3) #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED) #else #define _PAGE_KNL_ERRATUM_MASK 0 #endif @@ -112,9 +112,9 @@ #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) #define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * Set of bits not changed in pte_modify. The pte's @@ -123,7 +123,7 @@ * pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED) #define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL) #define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX) #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) @@ -187,7 +188,7 @@ enum page_cache_mode { #define _PAGE_ENC (_AT(pteval_t, sme_me_mask)) #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC) #define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER) #define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index 11eda21eb697..e7665a4767b3 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 533a327372c8..35f01203a14b 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -5848,7 +5848,7 @@ static int init_rmode_identity_map(struct kvm *kvm) /* Set up identity-mapping pagetable for EPT in real mode */ for (i = 0; i < PT32_ENT_PER_PAGE; i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r = kvm_write_guest_page(kvm, identity_map_pfn, &tmp, i * sizeof(tmp), sizeof(tmp)); if (r < 0) From patchwork Fri Sep 21 15:03:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610239 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C836C5A4 for ; Fri, 21 Sep 2018 15:09:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B97F42E423 for ; Fri, 21 Sep 2018 15:09:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AD9002E42E; Fri, 21 Sep 2018 15:09:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5BC422E423 for ; Fri, 21 Sep 2018 15:09:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C83ED8E000B; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A04958E0010; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E731F8E0010; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 686988E0001 for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id m4-v6so5798997pgq.19 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=o0aMUKOZelUiV25Gp0g6TuqJpMi614ruDMctxgitTNkYSPlzbJfHgXlydcJRLMC6wv 4k5++o99dREft5MWnb8XnS+DrdPsyxv95w8LH21iwAO4HAwyzqvJOIhzoJQ49Cg/p+Dy UsKJdgFEFAcux8dvamBSAqeRbligSCjrtC8xfWaWeMQnTRPTXfHACKAD2EvpIApJOges Vy36Mi9od2XRqjxQFeoRzy/ZQobkZ9Zf74WfqYmoX0nB+26lHXoW3L3b3eS+l7G7YUF+ AzOmMxwXVI86gDlnuYOPWETaEIe3394AHUUErGstSEQS/7sqqpSn0M/RyQ7qkBo90mIF /fLQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Caj4VGRKx/U+z5DaJL6I6SAlGjiBxUbkI+2jdVzCDIDnxF0APp dmdlgQwrKIWcPbLwI3ukgeFBES8IvnXsbVv0VjMfqr7Av5Ci7fjkmGc1WHRZ6tofpN1OCE/sb/+ BQnzwT+f19tWWOX8MGWlmmtHkbZBocZV8EliqxhO+F5Zq60kShdvfWyY700Y5yfMJAQ== X-Received: by 2002:a62:c581:: with SMTP id j123-v6mr31710020pfg.84.1537542531083; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYm7BCYUsNOP36R7PJi4bgq0Ibf7laLh4Bq23XbuQic30A60hs84KQ+ErF36N1L/u463zXV X-Received: by 2002:a62:c581:: with SMTP id j123-v6mr31709965pfg.84.1537542530223; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542530; cv=none; d=google.com; s=arc-20160816; b=Hf0Lmd1eeE3uW1NId7MI590NeDL5oNEIy3WL3nfkX8jE2US/5OjSw1I3ehzj+EBLQ5 8jTaqdILayZDYaZkUQ6EpqXAYMuLnUtcerBpEfYheK8j+uckIGT8TdNw3S1GffPQjD32 yDgsvT0FigZ+uHUecvPTEmo51lXYvOC2B7lD1RUiVgf6HDzGffLAgG1DzbuPoyWVSgN0 54frOnoUY0pyzhohLLdF1jz6+WQzLaDPDeg3EgfFtXmJn9m+ya0bQx3BKO3PIwHJ6HBy LGb8lqBUEUQcLWaRQSOiHLyHe5grEmx8crz98KlAqCa5/OIXwFIhyYFjT+9YiVXkXE7j lAtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uEehs0O2dwcwadl6JEivnjVZo1VhN/AB3p8NrI/AVMM=; b=atWoqiu+pLEG+OLO+lSnyQna08RSGfEwPVPO6j64ZNuvjqwJAQb7UD9slG6gzoznjG iNc7ZevkLk3RYCGanwyvG0olTZo3pxw8BySXGYVwsH0GTHEmHsVO/U/1yWmxtxWM1Hz+ 38K1E1gqurdaLpI8U7XhKQHqLpiaidDgect4kfLhBYCnHWJfiKD0/6w8C8ZlaxQYXDWp RYCSyFjEPwzsrlnUezSF/KkfprPcVUgPqjyo6PNhFIuF8nykBZ5jfrtWkPXGVEEdxb/f XFWUpCCKIl5zHkLbnIAF3RNJ2u6hHsSpavVG1ocLbl7P5IIr+VZ4tjnjrqgF2ZTbhUN5 7zmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id g2-v6si27080525plq.242.2018.09.21.08.08.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:50 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856558" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:48 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 10/27] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Date: Fri, 21 Sep 2018 08:03:34 -0700 Message-Id: <20180921150351.20898-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Update _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry(). In order to support Control Flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW. Signed-off-by: Yu-cheng Yu --- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 00aad8164dec..2d6ba1462dd8 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1170,7 +1170,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, } /* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse(se); ops->clear_ips(se); From patchwork Fri Sep 21 15:03:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04F6D15E8 for ; Fri, 21 Sep 2018 15:10:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E41C62E4B2 for ; Fri, 21 Sep 2018 15:10:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E15B22E4CE; Fri, 21 Sep 2018 15:10:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DED292E523 for ; Fri, 21 Sep 2018 15:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F4958E0019; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6EB2C8E0022; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52DC78E001F; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 0F8888E0020 for ; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id x19-v6so6720825pfh.15 for ; Fri, 21 Sep 2018 08:10:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=TN0NIUFY2ogdDNbPm5RxIBpiTqPq6mraMYNJaCbUbAA=; b=JaAZ2IxziNRFV9Vz7xz78gxybikn+RC5K8G576DQ50+EVC4akGEIg2YUgkI9PWsprw nqM1RbfIyIL8+/zXF9S7WGIBUDHPS+Nm7nSxABrOwcpUz5VgRYJR9HozHE+q7kNB5qky O/Sz8P9yk/sXcwgqw3qyTHsvVHc2vZ481AWQERHnk8iee33Lxdgwxi9Aw4qQ925AFFSL E/bMUnMEEq9ppvMROXvP5Bba424KZvO02wbKvEahlbsW9GT98+zneoK6q1nCjax1cZ+1 rLBhigqwEI5VY1rum126ZEhZ/wo+hAg8/hkC1o/LEvHi+trgvBmzVtLnCAAy2+OLwlZI GFsg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AJuC9BqK8n5dwaciV+ATBEFLCYGOUB9C01DmHwDedKKR4FY9aB T3dVHkrAxl8CsnVlXK5wNGVYFu96Ll0lbJZXinws6itBkAN10v+bikKg8/Lb/jB5PPbO3GQFxlR XP0jzECICxBnqcNxCSyFjasesR3MmNDqRk0uJpvqLJW7+4g5G2u/5M+brFaCbdfsfcQ== X-Received: by 2002:a17:902:900c:: with SMTP id a12-v6mr29217554plp.104.1537542627724; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaETJBTCu2bjp8KiT+dfwsws8l+mIi4fDLtOUUu72wbvfg7W13k8quan1H8aAHQc3Yyoq1a X-Received: by 2002:a17:902:900c:: with SMTP id a12-v6mr29211025plp.104.1537542530365; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542530; cv=none; d=google.com; s=arc-20160816; b=pcf/9cOPyIwL9laWTSWx4EvTtWTU37i7jPLoot3brbrJVcpocRbwXjYOMeyxb64woG y6pI7YKrMiPTAz6AQzd4Fo5fdD/TNCAbB+f2hJokfzjQq/wLn+b8K1ew+Xr4NMMm2CCE vkjEZX4xmxgwJrswZkDOxWeU9PK6kU1+Y41O3ECzPHVBEDGyZTOkgLgr2SP+wn4Qvc8j mktnDTusdAKRFY75SNHvo0fpFraUXxrO/RETzOBvDC7tMeMzuu4ryGiysjkJIP74pjgJ diHlGrvpr2O83MRU3soXvg+P92XLXQF9DUZOqkb5giFSfbG7vThWBpERG6I92Tw4UOLC r7CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=TN0NIUFY2ogdDNbPm5RxIBpiTqPq6mraMYNJaCbUbAA=; b=uvGenk7hEv22KPpPpElReO3gSd8Q64zHczUryNfg1sXI5iX21U0it0h1zrFWucFkX9 +I8sQ85+5ZYYJNJ5Bw5gYlcYI01wMRm16tHVCg9+FnhJv0rS3qWAaEg6iX7HDz9wtwVz fpZLqgDOfG3Qwhkh9NVMUVnBeV2Ur3RBCh6cca57Ha6yR2ZG0jAlIiBdvZk+ZhVPCO+E VDWIFDPjoLgjcyG5S8aFhKBk7Yb7RVBU/yICAqfyNhMwxQsgyYzuGQlvIxSCrS4grHHE JRCc///OXfZsnJgwDa+PBNXse2hHNi6i1V911n0PjaaWEu4qL+juP7O98iO2x+8HNamb IDOw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id z1-v6si5733063pfc.97.2018.09.21.08.08.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:50 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856562" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:49 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 11/27] x86/mm: Introduce _PAGE_DIRTY_SW Date: Fri, 21 Sep 2018 08:03:35 -0700 Message-Id: <20180921150351.20898-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A RO and dirty PTE exists in the following cases: (a) A page is modified and then shared with a fork()'ed child; (b) A R/O page that has been COW'ed; (c) A SHSTK page. The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings: Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW) Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs. When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 14 ++- 2 files changed, 121 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 95c918ad84ed..3ee554d81480 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -119,9 +119,9 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; } @@ -143,9 +143,9 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } static inline int pmd_young(pmd_t pmd) @@ -153,9 +153,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -294,9 +294,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + if (pte_flags(pte) & from) + pte = pte_set_flags(pte_clear_flags(pte, from), to); + return pte; +} +#else +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + return pte; +} +#endif + static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -306,6 +320,7 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pte_clear_flags(pte, _PAGE_RW); } @@ -316,9 +331,24 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { + pteval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + pte_write(pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + pte = pte_clear_flags(pte, _PAGE_DIRTY_SW); return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw(pte_t pte) +{ + return pte_flags(pte) & _PAGE_DIRTY_HW; +} +#endif + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -326,6 +356,7 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pte_set_flags(pte, _PAGE_RW); } @@ -373,6 +404,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + if (pmd_flags(pmd) & from) + pmd = pmd_set_flags(pmd_clear_flags(pmd, from), to); + return pmd; +} +#else +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + return pmd; +} +#endif + static inline pmd_t pmd_mkold(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_ACCESSED); @@ -380,19 +425,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pmd_clear_flags(pmd, _PAGE_RW); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { + pmdval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pmd_flags(pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY_SW); return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_DIRTY_HW; +} +#endif + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -410,6 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pmd_set_flags(pmd, _PAGE_RW); } @@ -427,6 +490,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + if (pud_flags(pud) & from) + pud = pud_set_flags(pud_clear_flags(pud, from), to); + return pud; +} +#else +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + return pud; +} +#endif + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); @@ -434,17 +511,22 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pud_clear_flags(pud, _PAGE_RW); } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); + pudval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pud_flags(pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -464,6 +546,7 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pud_set_flags(pud, _PAGE_RW); } @@ -595,19 +678,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); + if ((pte_write(pte) && !(pgprot_val(newprot) & _PAGE_RW))) + return pte_move_flags(__pte(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); return __pte(val); } -static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) -{ - pmdval_t val = pmd_val(pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; - val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); -} - /* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modify static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) @@ -1159,6 +1235,19 @@ static inline int pmd_write(pmd_t pmd) return pmd_flags(pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + pmdval_t val = pmd_val(pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); + if ((pmd_write(pmd) && !(pgprot_val(newprot) & _PAGE_RW))) + return pmd_move_flags(__pmd(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); + return __pmd(val); +} + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 0657a22d5216..f47bbc1f9c45 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,7 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57 /* available for programmer */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -34,6 +35,7 @@ #define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ @@ -109,6 +111,14 @@ #define _PAGE_DEVMAP (_AT(pteval_t, 0)) #endif +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ @@ -122,9 +132,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) /* From patchwork Fri Sep 21 15:03:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67D7915A6 for ; Fri, 21 Sep 2018 15:09:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 582672E429 for ; Fri, 21 Sep 2018 15:09:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C7122E42E; Fri, 21 Sep 2018 15:09:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 303912E429 for ; Fri, 21 Sep 2018 15:09:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F346A8E0014; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF8218E000F; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89C0B8E000F; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id EE83C8E0014 for ; Fri, 21 Sep 2018 11:08:51 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id 43-v6so1507621ple.19 for ; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=fbIU10g+3FIH2F9V39N52dFDZKOYUCa0+VKpaAcOC2w=; b=SLUpiY7OiO1nMvyva9W7NmwEEMOV6jz9fB5TI1XicrzqNLNGUwXJ0XrHCyQy+Aiyqu Ke+jfRRKrN/JTZoSCILJVdcqnIFOwSlWTsvZx0QaduhYbfO0cuBcRUK5KVxveA49XWTP W8sypZmqimDJExANn71vsphziLL0oLmRcRsEIy/8cw/3SwxyHCTDJ+FX/8z+kWff4HZc Fqdt09Q70YdDv30ZWLhASezwRfCNyooVp8ep3RMFcrdN9gJ0G+DgwvL6iQ4zEaju9512 0eC/MQASaRvQN56MV2vpgS/6FWccYXiShDHc0y+FUshjjA2Xx6yvd63qzlpuA8jMbRqw 5oFA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BCeOZZfcw29qvsvQI9ABkZE2VeHOOwGffFRh9L+4AT9T6RuqGj ndArLTS3fgzB8b2Nr2hcEYYvWTtg5CubI2hGZJlLaVm8Swtxu3ZokAzfcH8Jl0kqJZgfLQZCKhT ZmLjV4xkExwLok1buj7U06mJy2wXIm9wP1EWQjXiVEdUYDikY7iTrTjb4Mw5pfEkwwg== X-Received: by 2002:a17:902:b08f:: with SMTP id p15-v6mr17691336plr.296.1537542531645; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdbaxhidhub7zAlvQK0QCQsVwSaS+IvurdewiT6ZDxidy30pIfN96oxDIMdqWE4tIdbRcfoU X-Received: by 2002:a17:902:b08f:: with SMTP id p15-v6mr17691265plr.296.1537542530409; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542530; cv=none; d=google.com; s=arc-20160816; b=Jc+/XjLGJaMoI+WTKjPtEhvKo5UwlSh3wIKj45nw5KYGWIU1jHz8XTj2m4Rbtj5nhX LDwAHajCnPE8L5kWvCLxbH9vTqL7ahqaUAbW8wMC/0JIPbt+7h4GM0EqXqGEAtnW9wnC bOZkfFldq/u1YPFYHPHd6VMWLkxpLnvVaKZ5sIsXPvET9YNlWJSIPCId73dfCwzAnVDZ MNb4CwvTxvwaCDQd4FecqUzEJKyUwJLR/fdvzzV1Ca1yeaS4VwNnlFr6cSCCzxh1amHo pQt5TmoAplQDP7W+Wwwss8R99ry8h74a8PQvgjl04kL0zvv3onmJ8a+rt7qxygfp0Ml4 QAkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=fbIU10g+3FIH2F9V39N52dFDZKOYUCa0+VKpaAcOC2w=; b=X7Lx1V4xk27dsY1oyYcbA+6dVg7MKzgq9L0/GxqVYLLhpYGKpQVnxMToXAsXa4XvfM 3wQw4SgdWYrFvvkT/DoQDls54ETtcq6aU5PJr3Pq9ior/5KwyU2krSRBO1UrZ3t7p1RQ 0bliJLMNljdKefo4shosSr2UJGPckPviVfk51wI68b1sQuxJyXr0vctvXVzO+oPlbtHB rRb6ACCVFv6XdE87XrFEhaO9XWdZAbOKYoD+i11S3REGjeRfso+NSWGLHNnz1/LCYVeB lrPmQSJUkXdSSnBbAKphvVRTFQW0zQ5TM6dUJpgEsN1c4seJofmq1K8kLsqOExEUUs3y nMcg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:50 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856565" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:49 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 12/27] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Date: Fri, 21 Sep 2018 08:03:36 -0700 Message-Id: <20180921150351.20898-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When Shadow Stack is enabled, the [R/O + PAGE_DIRTY_HW] setting is reserved only for the Shadow Stack. For non-Shadow Stack R/O PTEs, we use [R/O + PAGE_DIRTY_SW]. When a PTE goes from [R/W + PAGE_DIRTY_HW] to [R/O + PAGE_DIRTY_SW], it could become a transient Shadow Stack PTE in two cases. The first case is that some processors can start a write but end up seeing a read-only PTE by the time they get to the Dirty bit, creating a transient Shadow Stack PTE. However, this will not occur on processors supporting Shadow Stack therefore we don't need a TLB flush here. The second case is that when the software, without atomic, tests & replaces PAGE_DIRTY_HW with PAGE_DIRTY_SW, a transient Shadow Stack PTE can exist. This is prevented with cmpxchg. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 58 ++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 3ee554d81480..b6e0ee5c5503 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1203,7 +1203,36 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pte_t new_pte, pte = READ_ONCE(*ptep); + + /* + * Some processors can start a write, but end up + * seeing a read-only PTE by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PTE and flush the + * TLB would be needed. + * + * When changing a writable PTE to read-only and + * if the PTE has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PTE is + * not a valid Shadow Stack PTE. + */ + do { + new_pte = pte_wrprotect(pte); + new_pte.pte |= (new_pte.pte & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pte.pte &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(ptep, &pte, new_pte)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +#endif } #define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) @@ -1266,7 +1295,36 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pmd_t new_pmd, pmd = READ_ONCE(*pmdp); + + /* + * Some processors can start a write, but end up + * seeing a read-only PMD by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PMD which looks like a Shadow Stack PMD. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PMD and flush the + * TLB would be needed. + * + * When changing a writable PMD to read-only and + * if the PMD has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PMD is + * not a valid Shadow Stack PMD. + */ + do { + new_pmd = pmd_wrprotect(pmd); + new_pmd.pmd |= (new_pmd.pmd & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pmd.pmd &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(pmdp, &pmd, new_pmd)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); +#endif } #define pud_write pud_write From patchwork Fri Sep 21 15:03:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC8DC5A4 for ; Fri, 21 Sep 2018 15:09:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB60F2E429 for ; Fri, 21 Sep 2018 15:09:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF7302E42E; Fri, 21 Sep 2018 15:09:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58EFA2E429 for ; Fri, 21 Sep 2018 15:09:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C61D8E0015; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6C4A88E0012; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2BE88E0010; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 1013E8E0001 for ; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id a8-v6so6304796pla.10 for ; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=k4I+IBksg1YUf85BSJ9dR9AdYeEvkDMb/4N/mPBj2zE=; b=ESjg729CvAaFBDRVTop5gyDEmG0b5luhFlBWmiDvo8A312UfTEMBY4LyMC7DS5qVhL f1AQWN5klkrQRBDxgCebNJxVuCh2i/Ej7HLQaDXtz9Xrjvn/ZUuW5b1twMIHU3ZFByB2 DUt9Epnc+75eQepLUwAUJPGOPaAYLAngbuBuchWk+Pw8gtQ+v5KQzQ1REFVrjunPamRk uWwFm3fyLZblqMI5AuixobDd6mAguipQuhqFkmLBLF+Be3MUxV/dxGb8pzRBHwmYf/ON 9Dz9Rm+/hCEkz+za6KO0PmxHrotBs1rioLXvAggiP1sbKWtjSGDI/LfT/+1f1QheK1/9 mFCA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Cqtz08wZV8erW2l/p+AGwe1qHxbqTYXIUPRi1fys/UYM6TKrD+ UX6NiVM1+eYWcZofNilpHPHxlXn9BQg+W33KmqMAHBzeGnGQpDqkXwaq51oxXvZRvsMxeOx9v0R i6E9SJjAEQ9SDtPBu6U5651zUT5li1b8kmzAeLNx0wfg9GGSlZQXR+rM5ZRVJxHwNyQ== X-Received: by 2002:a17:902:4503:: with SMTP id m3-v6mr44490251pld.168.1537542531754; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbFe48cYjYanIicFiqSaXv3WrlccdV595veCEzNppA5QrFlPzq/nDIGDyps3/uY2iSwHlkZ X-Received: by 2002:a17:902:4503:: with SMTP id m3-v6mr44490182pld.168.1537542530741; Fri, 21 Sep 2018 08:08:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542530; cv=none; d=google.com; s=arc-20160816; b=FwjfUY/JyIqID4SZ9xF6uy9QYnUSp1mJfyg/RtSQ3ZxN9E2+SUiDJaNACVn42uW7ja IYebqizXutDoFMHquhs7fVqik0I9BFDInOSB+delJ7pOcxC5lXsO0cmPL6JC3NTPJENi eVYCc7q2nj6FnNGUDaLyF2vfk/ymXcDveW0+nf4sE5/TSd9c/CLMCXTXKfqKC1Y5EcJc 8X6FdVuaUBQje7mwaCSGwavaYjSLlEZ2epSPUASXiljkjhX7eIpF/pZx3xfvTUi1d5Is plCe+LKCH1R970aI9GR3xQrPkzSA9HrI7nE4S16/pnvK2J8nj6vybkwYHLUyvkln/+H4 5NjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=k4I+IBksg1YUf85BSJ9dR9AdYeEvkDMb/4N/mPBj2zE=; b=IfGvq53dyM5jYG6PaAQ+jcM3euMC/6yN96AnniOnyRqPJERO5NE56OGv3XosHS2XhQ o4pDqJXRtYgoHU/h05RrhZ1dxhjAnXQXE1+U0EJewDy0r6jMNIBGcbHOs186lV5q5At9 u2jT9n/rSnWojoxh3bmrfBQBQuxnnrHr33pCxbpaOlysLD8FthX6ey5fn6+eAsRPvzEl IC06uX1TjNlFg5vfF3L+o9ZugB3KW7BnqHP+/C8icpEpfaTWKF/msg1s+wfVmsjirnwT OAI/jztv+H2Ope8Zhh7PQYAtPgtU0H9WBgIAm2fvI3vwmBb+sh2a5YrvVFWFdRpSGYSb 9VUA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:50 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856568" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:49 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 13/27] x86/mm: Shadow stack page fault error checking Date: Fri, 21 Sep 2018 08:03:37 -0700 Message-Id: <20180921150351.20898-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set. In access_error(), we check if a shadow stack page fault is within a shadow stack memory area. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 5196050ff3d5..58ea2f5722e9 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -157,6 +157,7 @@ enum { * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code { X86_PF_PROT = 1 << 0, @@ -165,5 +166,6 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, }; #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 47bebfe6efa7..7c3877a982f4 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1162,6 +1162,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Verify X86_PF_SHSTK is within a shadow stack VMA. + * It is always an error if there is a shadow stack + * fault outside a shadow stack VMA. + */ + if (error_code & X86_PF_SHSTK) { + if (!(vma->vm_flags & VM_SHSTK)) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ if (unlikely(!(vma->vm_flags & VM_WRITE))) @@ -1300,6 +1311,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * If the fault is caused by a shadow stack access, + * i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set + * FAULT_FLAG_WRITE to effect copy-on-write. + */ + if (error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (error_code & X86_PF_INSTR) From patchwork Fri Sep 21 15:03:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610251 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B64B5161F for ; Fri, 21 Sep 2018 15:09:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5FA92E422 for ; Fri, 21 Sep 2018 15:09:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 970E82E42E; Fri, 21 Sep 2018 15:09:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E0D82E422 for ; Fri, 21 Sep 2018 15:09:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E71358E000C; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BD5458E0018; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6567C8E0015; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id AF6F98E000A for ; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id 186-v6so5744885pgc.12 for ; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ShIZJSccJRhWxU3O525fNsEEmFvJS3t11XKEl0w+OJ4=; b=FdIKK42zRoqurJIh44k7o1zP/aZhX3YiCFKAEwgMiEvhrLw7PlvHCPxOdV9M5VHOW4 rkMggGvsk0Sw1kmojJqSn5kOnAwxMM+lzUOAYSV45TIPrydKMPVuMCG6JpoaRJQu9xce WH8TjMEpmXjKqNpN17Urpen9p83ukcW7tDaK+PVrsIHyRNWgTKVc7ZtLQC76vXvMITvG CUped8Vh6vkT4QmbovC0Q946AcIEfQ93YkrJeXxhTlhlx7STsyMwd5ALKWmrVbqwAErQ aBUb4MCl4mT705FWqkhu4hJIvv+4hPHGVYl4HC+/BAdWsEXjdvPXnSM4WrkzqXtob/p6 mbqQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51ACK6Din7wkbkx1vsFJgHThTMeOc0bfdj0Uk2vhOL/yFbOtrJ/J RBzgaocdIFuT58NWViMwIPofI7atNpEVjRiV6yvXiWn4FGfwZHRBq/jgtU6CX9XvbxKqRH2A5BV Fw7lEhWnnnue9u5FI4T0RN2Yrh7wsF6JZ+D4X3U2ACsoqDeu8yCwaZCZSj1e1NhEOow== X-Received: by 2002:a63:115f:: with SMTP id 31-v6mr41881948pgr.53.1537542532388; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda8PpP8AA0x0Z0kgffBCuvZW9CCE5DAafwsrYTaC5SIjMZw8EdbByVN0tgnDVA8EDrNKBYR X-Received: by 2002:a63:115f:: with SMTP id 31-v6mr41881867pgr.53.1537542531091; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542531; cv=none; d=google.com; s=arc-20160816; b=HnNQdg/vW1iwnQ9UEXDJKgzM/YJyvWuCSmKhAq8on0x42LnScKDAzqHw2plMXZZmJc lRRPFc0K8TzJ3X7TPGoLIgsrWp2yHEp9Zx7V4pdCiQ5kKknZGHfjshcoWZL2Y7zR4KWF RjbQfUQUyaxNx9tDqS5TqBwCGk8ylPTBmCMM+nJOn1oLLnmqQkzw9MKglFEccDJITW6F JaQe8Cbc5Sicx5ylZprfiUJaOUQqoWJybI4cmVmMsz5y8psIzsDqZ6F5xcny1YGrz3t0 zIk5nRiQcTA446mEKz9zD/LTUtzqLdHSRYjUEnlOVLDB7tKRs4Hhs4aKJSsNYUvpdK4G Komg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ShIZJSccJRhWxU3O525fNsEEmFvJS3t11XKEl0w+OJ4=; b=xnAayK1hVfKrhe9s+JwD53NaGEPUL0EoGHmzWw5Y3Yn6eIKLJfSW2z4Agk7KD3CdwQ LQJOnGSK++eck7AqnVzC7FXZs3EMbfftTcyJh+gBwOyepmhZJQJPYQnjB19OboJYKMmQ wHE7PoGKzEd5glmoDyGcOyGvrcBWKMxcEFp1pRKh6uqe+Un2i3Cy+fWAEucdZXqrVOqz qTm2vnPqKXA5fqywOADDC1RHpcCD2TC606gd+MVOWnv+3VUmpzzJifF6pUz6ts+a4hCL JDtr+i8ZwxLB1B8MP6Dl50G04vXHMZBCREx7NiV/agYteCnuqIbgAaNpbQWIphESHiX5 BJow== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856571" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:49 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 14/27] mm: Handle shadow stack page fault Date: Fri, 21 Sep 2018 08:03:38 -0700 Message-Id: <20180921150351.20898-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a task does fork(), its shadow stack must be duplicated for the child. However, the child may not actually use all pages of of the copied shadow stack. This patch implements a flow that is similar to copy-on-write of an anonymous page, but for shadow stack memory. A shadow stack PTE needs to be RO and dirty. We use this dirty bit requirement to effect the copying of shadow stack pages. In copy_one_pte(), we clear the dirty bit from the shadow stack PTE. On the next shadow stack access to the PTE, a page fault occurs. At that time, we then copy/re-use the page and fix the PTE. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 15 +++++++++++++++ include/asm-generic/pgtable.h | 8 ++++++++ mm/memory.c | 7 ++++++- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ae394552fb94..57eeb2230340 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -872,3 +872,18 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_mkdirty_shstk(pte); + else + return pte; +} + +inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) +{ + return (vm_flags & VM_SHSTK); +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 88ebc6102c7c..b99aa3677350 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1127,4 +1127,12 @@ static inline bool arch_has_pfn_modify_check(void) #endif #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +#define pte_set_vma_features(pte, vma) pte +#define arch_copy_pte_mapping(vma_flags) false +#else +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +bool arch_copy_pte_mapping(vm_flags_t vm_flags); +#endif + #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..1fb676ec7da2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1022,7 +1022,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * If it's a COW mapping, write protect it both * in the parent and the child */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { + if ((is_cow_mapping(vm_flags) && pte_write(pte)) || + arch_copy_pte_mapping(vm_flags)) { ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -2462,6 +2463,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2535,6 +2537,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -3045,6 +3048,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte = mk_pte(page, vma->vm_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); + pte = pte_set_vma_features(pte, vma); vmf->flags &= ~FAULT_FLAG_WRITE; ret |= VM_FAULT_WRITE; exclusive = RMAP_EXCLUSIVE; @@ -3187,6 +3191,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_set_vma_features(entry, vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); From patchwork Fri Sep 21 15:03:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5BCF161F for ; Fri, 21 Sep 2018 15:09:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C64B12E422 for ; Fri, 21 Sep 2018 15:09:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B9D902E42A; Fri, 21 Sep 2018 15:09:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37ECE2E422 for ; Fri, 21 Sep 2018 15:09:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C44968E000A; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 852628E0016; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EBEA8E0016; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id B99F58E000C for ; Fri, 21 Sep 2018 11:08:52 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id e8-v6so6309296plt.4 for ; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=TL1uOxD7TZ4uTeCTIe1f+ZtRmUZ8eyJkPEADKHoouHI=; b=E9Vn08bM0YzLHXNOEMHoDj+FSt/b3WyfTS4M5wxhKe9URD6hvyKz21CHtNERYOVrEj 0xW1Q68sMne3UL7AeJCrQEjN+jhK9/ipXBkcWmAH5jkrBFs52bzXd4EzWZWdGPP3saoH vHm1+QndW6f5oCphXNusCeQZeZh9jGPV90KBjqVoxSU5CL/Xx689xL4Nh+wlRDOqoAht sCz+IKGL/LZBtoJB+km63YKuNwdvsQeFge8ZlSD5y2cO5akYRD5A2y4yIpguWcltjnq3 qSYCi14JvJC6fBFuHAOh11+izaCHz87aHyYeZ9cQAFouoeAR2ifKknpsH8qRRfJ+r145 fSpA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DWJMf7AtLyEpjKC6ClwgN+zVxUwRNLzdKU6r9m2jBXo90NjUPS 2IX4VcEwmhjBR647w5gE+c3aadl+qwTcki5RhCJZtXau5wmPD1i5TkXjadawIrp7YJIBWyzG2ig f10tT+P8MDLS0uzJWLacRtVgXOwKzUyasTqJTc9yoGyf8PlLVc9Yd2IVoJCXqgztnPg== X-Received: by 2002:a62:5d89:: with SMTP id n9-v6mr47052846pfj.102.1537542532440; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdb8lh8Ka3zNwQeObq9ghwumZ9Wv+dDS4ceHxDdZDcZ8dpgeuHcwMrT6iiiRnXRfnNUYal0k X-Received: by 2002:a62:5d89:: with SMTP id n9-v6mr47052773pfj.102.1537542531482; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542531; cv=none; d=google.com; s=arc-20160816; b=mTifYmqVpk4zjUCld3Lf8cs05q2aQst0vH2gdBDkISbSX2jd16BNRXFDTHCU8I2Tl1 +xzzZo9r/IxerYWSKAZ/6FnfcMlXe9nPXqngelD2DkN+GedYgaltIfNjYE0iIKQe2n7g +d/RKWhv910wzLD9wsS02tI0JzVtHGzoFcRZNM7IOBHCNSq+HA2kEJEfT+wLzEd6whEh 2Tqh8abmVB08UJ2WZLEXpTeKGv+p9wmjMLBqGZA33RVVNL08XHMr4HW4zubjAElz9HhO 1TVMoykM7a3T9jgZiFMyyiEK2padXnlrHpBAv5G3JXCWrY31NqpLW8XbIMtNSdtx7xw7 NAaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=TL1uOxD7TZ4uTeCTIe1f+ZtRmUZ8eyJkPEADKHoouHI=; b=OoqV8+SLk5rCNl2ZevwJZUXy/RpGGMlh4MOaCycNtx9AY3/8tOVI2BmqX/x17KGRV/ IRW6nJfbgNQEp4Iw0Xju1PPBynJgyVBs27LhgvXXm7il1MxqQ1uKP1MfUmn23CVddWl1 cSpoiplc3736YRCsPg90YM5ute653KTzFrvkF4kp5qVwiloAQ+a0xkVbPccQnuXK9e9/ 4xcHZA+vfdcXhRD49X5F+H9VwL8fUHk21Hsktn2bbcqpGHbRM2XYPdXg6FjzhFWjfTy2 Sp/0iXjXo0OntmTMDartEbtshXGJ1cYxbHcEYFk3kbCE7uKbjxl/vY2LM64/ju9Jte9r 2zSQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856574" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:49 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 15/27] mm: Handle THP/HugeTLB shadow stack page fault Date: Fri, 21 Sep 2018 08:03:39 -0700 Message-Id: <20180921150351.20898-16-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch implements THP shadow stack memory copying in the same way as the previous patch for regular PTE. In copy_huge_pmd(), we clear the dirty bit from the PMD. On the next shadow stack access to the PMD, a page fault occurs. At that time, the page is copied/re-used and the PMD is fixed. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 2 ++ mm/huge_memory.c | 4 ++++ 3 files changed, 14 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 57eeb2230340..ccdfd3dd7163 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -882,6 +882,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index b99aa3677350..a91f07454ced 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1129,9 +1129,11 @@ static inline bool arch_has_pfn_modify_check(void) #ifndef CONFIG_ARCH_HAS_SHSTK #define pte_set_vma_features(pte, vma) pte +#define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false #else inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 533f9b00147d..df39ae20fe40 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -597,6 +597,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1194,6 +1195,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1278,6 +1280,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1349,6 +1352,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); From patchwork Fri Sep 21 15:03:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610261 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A11715E8 for ; Fri, 21 Sep 2018 15:10:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3915C2E42A for ; Fri, 21 Sep 2018 15:10:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D52A2E430; Fri, 21 Sep 2018 15:10:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91A652E42F for ; Fri, 21 Sep 2018 15:10:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A36D8E0016; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 607AA8E001B; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EC9E8E0016; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 5F8A58E0010 for ; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id 11-v6so837836pgd.1 for ; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=h73YAqb1qP/EQ0K+GtjIqsg5+zhUjTo3FsgsOLKvJKU=; b=BEMIYN8ExrDu5dYrwbuymFukbaJbbvyeCm2dJ0bnS3rhJmY60g4WuvnCoh75Ft6h9A QMCa4RMVXyYQAHdkJKz7qJIoixLea3sL7RB0/C1BCkkZrhWwqAXFV0C6/zC+BlNLB+9k 6X5Zbc7bmsLxElo+eBTt3tcFJiaOeMwDccd4Pbz3yH3oYYVJuD+C6akMZYHCxZRDrVv6 Ym3K/oib5MvOxC0sc6V2amw0HhdFULp+Nlzb+xGakL5KKuVx6ojTlorMgp/1isIs1dAl Ny/dZeXFFWlricTwwMNBc8AAhpwTVgNJQg6IkH/ssy+JlcbigkevqCAYd8uZjzAc6fU8 ZLww== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BNlgFTAteUzjSt6ItfHE7l2N2Ow7d+4f6kZwkly79C7na7XmSD gsw8mca0y/7ROqrXfuJlS/TXsoWlHDymp5bsVxzVgiIjoXT6SVuWCEuUcjKdJXF3nEKQFBXgnvH sDmfD/ACjSkqPpjDorwS5tRJhgErgQp/zPucs/dGaSCA8daSeirVcZ2/7smAq2LqXGw== X-Received: by 2002:a62:5ec3:: with SMTP id s186-v6mr47308137pfb.146.1537542533084; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYLlxP8s4Th6t8lCwfdYo4ebbD24AVSssliqr6w+RI8GsWPf+TAJRiqVcQcvQBzLEgsLq6L X-Received: by 2002:a62:5ec3:: with SMTP id s186-v6mr47308027pfb.146.1537542531518; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542531; cv=none; d=google.com; s=arc-20160816; b=0oCEV0/s2v8ebCvPh04uokOkO0PSrPum6htgkGZEQnnwIxtPdeW7herHiIEdFg6g2K 3boDcp0fYEsq9hhsXFMHy+XzrmEnQLgPuP7kw+LSVmJ71KOJsFO1ldFMMrM8EK1gdsYe 4jnAQrp+oa5z2N/+EOYcIKgrUPNbwEEUx1HLlWkqjmEOhjiwv6JFjnL4JlK2LwKTfxkQ m4QcG54jlVbX/lCHPJ6+R/15Jx0ZEiG6D763e3YobQS2P9lo7KgckqjfOeke55R+7w47 mmZFccdEKjH+aceAkVqHeLGWDmJZMt0VmqIahz+C2jOfQfJfEMJrj82nfYYBtZdp7F05 8kSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=h73YAqb1qP/EQ0K+GtjIqsg5+zhUjTo3FsgsOLKvJKU=; b=AvfxdPQo2twYRIsSXHRHkwZlB1rs4cPQ7OAZEX4vCDRWyJXN0ZWPoNfmXe+0RMB/jd 0TqdgLjBn/5GNe/UPYSOGuYlJi7hjcH53XIX9ic23SGgkIqxPx0aHsKWpD3cTYCh+EI6 82PIiN27RRHB+qmmqcu3mGHVIi3+msVFRyO2u9gZzDWabKLAFqbkE6WjLjh8/P5ybo4d y8h5rfQ5Cy8rQVl96tMB0nhuKgPUHxuQksoi7s+RyivkZI7BnxZM3GIZ8WriyL02lEzv sM/ZA+IdFbMW7XDORI+G8+HIvYggfIY4jAqcMcZLWH5rqga/f4W5PbyZWAYBBQEeKKco JdNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r7-v6si26956600pga.77.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856577" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:50 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 16/27] mm: Update can_follow_write_pte/pmd for shadow stack Date: Fri, 21 Sep 2018 08:03:40 -0700 Message-Id: <20180921150351.20898-17-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP can_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW. A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW. Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive. Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?" Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 19 +++++++++++++++++++ include/asm-generic/pgtable.h | 4 ++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 33 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ccdfd3dd7163..e13a020e37db 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -894,4 +894,23 @@ inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); } + +inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_dirty_hw(pte); + else + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_dirty_hw(pmd); + else + return pmd_dirty(pmd); +} +#endif + #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index a91f07454ced..6223017929be 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1131,10 +1131,14 @@ static inline bool arch_has_pfn_modify_check(void) #define pte_set_vma_features(pte, vma) pte #define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false +#define pte_exclusive(pte, vma) pte_dirty(pte) +#define pmd_exclusive(pmd, vma) pmd_dirty(pmd) #else inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); +bool pte_exclusive(pte_t pte, struct vm_area_struct *vma); +bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/gup.c b/mm/gup.c index 1abc8b4afff6..03cb2e331f80 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -64,10 +64,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte(pte_t pte, unsigned int flags) +static inline bool can_follow_write(pte_t pte, unsigned int flags, + struct vm_area_struct *vma) { return pte_write(pte) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pte_exclusive(pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma, @@ -105,7 +107,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; - if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) { + if ((flags & FOLL_WRITE) && !can_follow_write(pte, flags, vma)) { pte_unmap_unlock(ptep, ptl); return NULL; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index df39ae20fe40..c70aa8fa4cb2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1387,10 +1387,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) * FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags) +static inline bool can_follow_write(pmd_t pmd, unsigned int flags, + struct vm_area_struct *vma) { return pmd_write(pmd) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pmd_exclusive(pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, @@ -1403,7 +1405,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, assert_spin_locked(pmd_lockptr(mm, pmd)); - if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags)) + if (flags & FOLL_WRITE && !can_follow_write(*pmd, flags, vma)) goto out; /* Avoid dumping huge zero page */ From patchwork Fri Sep 21 15:03:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0B9F15A6 for ; Fri, 21 Sep 2018 15:09:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF8A42E42A for ; Fri, 21 Sep 2018 15:09:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C29712E42F; Fri, 21 Sep 2018 15:09:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E6112E42A for ; Fri, 21 Sep 2018 15:09:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EFA08E0018; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0B5AC8E0010; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AE938E000C; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 1330A8E0012 for ; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id q21-v6so6698647pff.21 for ; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=nuX3QM8HX4YtIzApwaJeEjWpSb/A/OxmLnTA/JbU0aQ=; b=pUp4eM5zJ2Pa85NdSgd+TJ8+Il2h4GE5owNrzkccwqvYEzlZQlC1wNfU277HTjHRpD xgFuUSyc9Qh38wvF4UrZH0KgxZMfGWJD/V2jKw+D9n1XRFRxBtOMlyZlCbxtFEdmua4A SvHcQcuDn4h5YT4f+Aw8xqCK01O/N182Yyq/72BlaMNyXxuWfNdoPFNJRYdzfAZzOXWz AA4iASDjstbHjHpRWznAkWkjiqBCJ8HzTV3xLWmiQltPG+ohNf15sGhdqP8yFXk5ZP4R I84EErCCcsgjQ3xQI/QBS2qUJfAqBwoCToV4zSDayMtK/AZow5NgZaQRYbZ0dSsVbL31 Dspw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BQMUbt7adItZV6qkvSmsvGDgERoT1hKhP7B6cehHJMmIfkyUM/ bNYrkwc5iQ41u+CPS8f2V7Jkmi2VC1vZonoVKIYEYn4q1fkL1M2NlZ0ddfpI1V5pD/oZC8zxAsZ TjNk6z4HcD2etuhonwA0F1rjAs8Z/E979s7y7SQfIJDJtt7vCrOug5EZs7yuS7vHCkA== X-Received: by 2002:a17:902:464:: with SMTP id 91-v6mr45432690ple.125.1537542532704; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZXlMph8WLtb5C9ijwUQTyITsD4LLDxfCeF3mqyXNFZL0wW7GBk0l+H7s7F4EYB3RjLelzc X-Received: by 2002:a17:902:464:: with SMTP id 91-v6mr45432637ple.125.1537542531834; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542531; cv=none; d=google.com; s=arc-20160816; b=aNNAGa2Xsc+JPnej+92wRqCszb3ZRMlILFjvo0oVRQ6vl3D6azCZGNqYeAp7qNyBfD erEcqgXgOpmBsI0ztDtWsbN4cSCqPlm3WUN9+Kzhc1SYbC144A3pmeA3f/cak2koSuEy SMcvbVxH5iPDoCWBmEcw/+bSIGPb+T4rDRg3APl5SbUlRv1YrKSYb5oPydTt8ZKUll3w WkZTjTTHT5faCKC8tL1XHKd1qeVSRUN/ejubLonBnTp01Xk0/VyNZqDSg/pdcpzGeJRC e30ttKzq/lv2uzXUSBkDjodAH2V3j6011SSc57pQ2OnvvXjY1eStIl/cXb688xD8kyN5 +huQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=nuX3QM8HX4YtIzApwaJeEjWpSb/A/OxmLnTA/JbU0aQ=; b=FHww5G1jjwG+cSUOU0enT+bS/o/b+VeiaOrSYjhMS9f6uHt7Qy2xsjnikDVHjSNTn0 dGe1ynh2LKMHVhqBAgsvofk4V2YaO49j9UJtVvIvoHPmaM/zvlFQQLzpS1AomJoVw8ep 8dTbHmKWzGis8z1cNVtFGZNdvV0dQpc9iWS44AcPADDF24tYJO/y3P6MNG9lplSPc+08 YzBP/8KM4B7EF6UGhuBGq8n9QxvWDnMOftWqIbocRb/rrtZF2YMwRTkwx5gwkOsF9AlG BSkFNECFkG61kpkbFyEwRX2JqlnuRc35hnyaH+clrehhPu5CYt7v9S5PGukTYX9C1XXR rI8w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856580" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:50 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 17/27] mm: Introduce do_mmap_locked() Date: Fri, 21 Sep 2018 08:03:41 -0700 Message-Id: <20180921150351.20898-18-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f40387ecd920..c4cc07baccda 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2318,6 +2318,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len) static inline void mm_populate(unsigned long addr, unsigned long len) {} #endif +static inline unsigned long do_mmap_locked(unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags) +{ + struct mm_struct *mm = current->mm; + unsigned long populate; + + down_write(&mm->mmap_sem); + addr = do_mmap(NULL, addr, len, prot, flags, vm_flags, 0, + &populate, NULL); + up_write(&mm->mmap_sem); + + if (populate) + mm_populate(addr, populate); + + return addr; +} + /* These take the mm semaphore themselves */ extern int __must_check vm_brk(unsigned long, unsigned long); extern int __must_check vm_brk_flags(unsigned long, unsigned long, unsigned long); From patchwork Fri Sep 21 15:03:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC1075A4 for ; Fri, 21 Sep 2018 15:11:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D81762E429 for ; Fri, 21 Sep 2018 15:11:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CBEBC2E45E; Fri, 21 Sep 2018 15:11:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E80352E429 for ; Fri, 21 Sep 2018 15:11:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BEAE8E002C; Fri, 21 Sep 2018 11:10:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 16EE88E002A; Fri, 21 Sep 2018 11:10:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05E6B8E002C; Fri, 21 Sep 2018 11:10:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id B8CF78E002A for ; Fri, 21 Sep 2018 11:10:49 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id 90-v6so6240332pla.18 for ; Fri, 21 Sep 2018 08:10:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=LQKs8u0lRoHwWxVp4ToodOHXeyMmcFV7IcXbsJAoWsY=; b=OPTCqaNo2s2rey6PE2uES2na34ofYEnn9K4wYbwHNVAVaYCy64CZH5hryY14/dlfaq dwliSiQBqonQqZuCSxW+zTl8Vl4bDycPX4bDot4osAAnk3gyjqCyiRXH1VL6GSHBWqTH j3vnq6Z8G0GYQNjD5T3NSJ7E2h0Pj6caFOF7gK0z4vcFX8nlOXIFNn5G1Duda10WL2AX wB/j/2eWS6GcLRbGj4M6C5X5GVTG5M38UG41ZEFoHOK4yrudnPBD/6YTzHNTpmo36I+1 IMt17iXlhqVKIv5tdvZmP+MU8A5XQOfLBKgWoYXNKvVHlwW252T4GJCQDszp0Vp1us/P atKA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51A8dNoaYpMC9EtwmK1QIlhCO2dy97ZLRp+7yWzOgefT2NXzKy1k 6gz/En76q1oJdXoKHaI5OlOUIlde/Z5Mvjub1FHyk3fNxhy2epdhDv6l+QhUdA0infJTKDLYE50 hgZfASPimz4d+gpwt3OgTXYzZLbKp5Utu5Ko4MqyvskNoU3f0cXKnWvghqGwE+xb9Ww== X-Received: by 2002:a63:9752:: with SMTP id d18-v6mr40057359pgo.405.1537542649397; Fri, 21 Sep 2018 08:10:49 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdblt72eMLVpxcfD987EVjySRthRU5ybezLtRg7d1K6ILE1x3YuxqsxnwdEVxtbWBWx2an// X-Received: by 2002:a63:9752:: with SMTP id d18-v6mr40049752pgo.405.1537542531841; Fri, 21 Sep 2018 08:08:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542531; cv=none; d=google.com; s=arc-20160816; b=oHFcvCy6Jj6rySt6cma54Ov5vTNAq5pWTIw1qd4tFAwuBPTlDCehNX2YMva9aTtGfp /shg1E768H9/iCS88o7UEea2ERqi9wEYIAjfHyt/BiTS+AT8tGlRVw2D0GmS5UghYpKc 4i6lZgp3AKByN78vWrw7/6koii3nQ/blgLlWsOUIwRoljqZufJWEftKZd4s0B92vMB93 8dmMxwX446Ai0IepnvzkNk9bBjQWOQRcOHbM7GqM+KBAiku6xSLZiP+3OahFnIKRJeaP 6qont9MsD3Du5gK+pTBWMg8EHkni6yd72OzC8Dfz8jOCRuOjhFr0BUnWhnfunxFo2l7p iufw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=LQKs8u0lRoHwWxVp4ToodOHXeyMmcFV7IcXbsJAoWsY=; b=dy9HGy0AM9sHb3PH5gHebuOQ2eP30A2jXN8HkEFxWhOl7hekxo7Si9PK2ggJAJdqI7 3JR2Lb9wm10QjEJwtZAiIkUu2XQ9tJf+oJoofjweGa2h3pKTm6AK6DkobNB1bz22cKZ0 iKZuK6t+g0oXCO+0C+mTpF6BqieP2TR2LCoOZbjIicdfoDnppBgJZ/H9u1MueSMtdTKO SSyd558m0yRGv/zXa3PSMCNiWzdwEawyHCNM1o1VQhpSX3Q95ggsQtJ3YyW72UH7RDf2 633OMMIFEHaBX+bCuR8jovEaAYF9muhg1tFUnmlVqs9LzOZk+atf52HFmdAkhebQgOVf 9//A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r7-v6si26956600pga.77.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:51 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856583" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:50 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 18/27] x86/cet/shstk: User-mode shadow stack support Date: Fri, 21 Sep 2018 08:03:42 -0700 Message-Id: <20180921150351.20898-19-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. The shadow stack is allocated to a fixed size of RLIMIT_STACK. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 30 +++++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 14 +++ arch/x86/include/asm/processor.h | 5 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 109 +++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 24 +++++ arch/x86/kernel/process.c | 2 + fs/proc/task_mmu.c | 3 + 9 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h new file mode 100644 index 000000000000..ad278c520414 --- /dev/null +++ b/arch/x86/include/asm/cet.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; +/* + * Per-thread CET status + */ +struct cet_status { + unsigned long shstk_base; + unsigned long shstk_size; + unsigned int shstk_enabled:1; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk(void); +void cet_disable_shstk(void); +void cet_disable_free_shstk(struct task_struct *p); +#else +static inline int cet_setup_shstk(void) { return 0; } +static inline void cet_disable_shstk(void) {} +static inline void cet_disable_free_shstk(struct task_struct *p) {} +#endif + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 33833d1909af..3624a11e5ba6 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -75,7 +81,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 4731f0cf97c5..e073801a44e0 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -777,4 +777,18 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0 /* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2 /* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4 /* kernel shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7 /* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8 /* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index d53c54b842da..63918cecf367 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -24,6 +24,7 @@ struct vm86; #include #include #include +#include #include #include @@ -505,6 +506,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET + struct cet_status cet; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8824d01c0c35..fbb2d91fb756 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..ec256ae27a31 --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,109 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * cet.c - Control Flow Enforcement (CET) + * + * Copyright (c) 2018, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int set_shstk_ptr(unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + return 0; +} + +static unsigned long get_shstk_addr(void) +{ + unsigned long ptr; + + if (!current->thread.cet.shstk_enabled) + return 0; + + rdmsrl(MSR_IA32_PL3_SSP, ptr); + return ptr; +} + +int cet_setup_shstk(void) +{ + unsigned long addr, size; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EOPNOTSUPP; + + size = rlimit(RLIMIT_STACK); + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + /* + * Return actual error from do_mmap(). + */ + if (addr >= TASK_SIZE_MAX) + return addr; + + set_shstk_ptr(addr + size - sizeof(u64)); + current->thread.cet.shstk_base = addr; + current->thread.cet.shstk_size = size; + current->thread.cet.shstk_enabled = 1; + return 0; +} + +void cet_disable_shstk(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return; + + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_SHSTK_EN); + wrmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, 0); + current->thread.cet.shstk_enabled = 0; +} + +void cet_disable_free_shstk(struct task_struct *tsk) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || + !tsk->thread.cet.shstk_enabled) + return; + + if (tsk == current) + cet_disable_shstk(); + + /* + * Free only when tsk is current or shares mm + * with current but has its own shstk. + */ + if (tsk->mm && (tsk->mm == current->mm) && + (tsk->thread.cet.shstk_base)) { + vm_munmap(tsk->thread.cet.shstk_base, + tsk->thread.cet.shstk_size); + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + } + + tsk->thread.cet.shstk_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 44c4ef3d989b..bffa9ef47832 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -411,6 +411,29 @@ static __init int setup_disable_pku(char *arg) __setup("nopku", setup_disable_pku); #endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) + cr4_set_bits(X86_CR4_CET); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk(char *s) +{ + /* require an exact match without trailing characters */ + if (strlen(s)) + return 0; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_SHSTK); + pr_info("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n"); + return 1; +} +__setup("no_cet_shstk", setup_disable_shstk); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization @@ -1376,6 +1399,7 @@ static void identify_cpu(struct cpuinfo_x86 *c) x86_init_rdrand(c); x86_init_cache_qos(c); setup_pku(c); + setup_cet(c); /* * Clear/Set all flags overridden by options, need do it diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c93fcfdf1673..4a776da4c28c 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -39,6 +39,7 @@ #include #include #include +#include /* * per-CPU TSS segments. Threads are completely 'soft' on Linux, @@ -134,6 +135,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(tsk); memset(tsk->thread.tls_array, 0, sizeof(tsk->thread.tls_array)); + cet_disable_shstk(); fpu__clear(&tsk->thread.fpu); } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5ea1d64cb0b4..b20450dde5b7 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -652,6 +652,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "", #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif }; size_t i; From patchwork Fri Sep 21 15:03:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610257 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4569C15A6 for ; Fri, 21 Sep 2018 15:09:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 348862D992 for ; Fri, 21 Sep 2018 15:09:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 285762E42E; Fri, 21 Sep 2018 15:09:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6F622D992 for ; Fri, 21 Sep 2018 15:09:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 450D28E0010; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3E4A98E001A; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1ECB8E0019; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 5DE478E000A for ; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id p5-v6so6676516pfh.11 for ; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=vXXsumCCPQu82rm5dgLsDNuf4LdMeyXLOXjYtSFPX3E=; b=S1g8EkSitBNcxLK7gPUCUnz+KKRwxNjVPM39QECqiDTHZRhxNguBYX66MhpRv8pGbd lHPHk4rcY/7nmtGZRoNp01LzRsVouZThK1OOI2zVJlD7bElmwg9D5Mpq6KLm3Pls2XoJ JpUUr0sszdKoEDc4E8HwvwieLwo7HEcBIG9zJGlkk6ne3iUhNrzcUd8KAENsOsMV47jO 6qn/OtJtY8zUn4UPzrsN60Ld8kuSWDW/GIrjV5dU7+xx/nrtNAVJWgDa+rZzDMTojdCH gTZ6P3HFwwJj9UHxtd0iNZDoSLl1q/es4GZKxv6HOXgqOfbN43Cqm15O1SmSbzFRCGR6 Y12A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AmI1hpYZKmUHgC++YJOpZbHHO2992NCEMv2CeNWNVbBrMELFi7 tWetogTVBAc027VJuFRJ847KRRHCi4I2f+/kavftx39uNTcPCFyQZTksstNop+tyTe2JtwLR5x7 zaCPGpQW5y2J3BMYFVJbWfoDYXPLK36ZZ3WdMVGtbPcDyzcXYSOsAPH6P08MwAe/znA== X-Received: by 2002:a63:26c4:: with SMTP id m187-v6mr6919465pgm.268.1537542533066; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda/yy4AQdETn/3RB+mmLBBRzVMvP9dbjn8vp5G5SDbq7+v1cS8XDZgcFXQIlcq8pyZnJpVU X-Received: by 2002:a63:26c4:: with SMTP id m187-v6mr6919413pgm.268.1537542532147; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542532; cv=none; d=google.com; s=arc-20160816; b=BtOdVyeiBaan2E4UNnqAv1g/e/aSRwWJ/xay+pQTDECpmAGA5yIOMUUXYOuIisdGyS i9Sa6JEIf4PdAEfZFmzwBfIm+bV/wInDvPnSEXqJyhbp4W9rBDmAWKGnou0yxkzJwpPo HlTqPbcyrw7DyjSpZCXmfjdKUjbLjum2jXO70L6tHPHCYB+dqyXEqWfEf5Zw+8x58+C5 /hegGelwQObsgkTKt+iJIlUz3S0iF62Y1RUeNkFK4mvZpwlUVxe028KgVk53IzrewkWy bKLo63hoYx1FL8+xC/ZisyfnZCVepMWsLi+H1RBHfIDma+IcYVb1SlRhbFhgJJDWA+vA 74QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=vXXsumCCPQu82rm5dgLsDNuf4LdMeyXLOXjYtSFPX3E=; b=Y97OVPdID3cduHTHkwfF7YtSFkyj5HiFCfIeS2A3gwduZ1ATRFtJSn1ZggdKTbXRYe vEsHxVJriW8PASqgWChtefj8ms86yQy7Ey3pdVHJP5Hq1zlygK6zhn4/2qMk5Rt+RJxF oq7mmvNmwQ1447pIXxDEZsBWJRvpFseu0xPZnKQDWWZhR4kZRbWu01JRcJYne1xK68Qp 0OdlomewuYNIeQReW3N9E4xCsmBqoqKgJ0TOdpVfBLNntxFHHUyQThkvZbyCmxXdfY93 Kj1idi45JP0789vBobCHILRQo0Vk2PdAhcMj8wU9Q4kSm8NZAFFYfF7GMp3fXvjuymiO XLvg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856587" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:50 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 19/27] x86/cet/shstk: Introduce WRUSS instruction Date: Fri, 21 Sep 2018 08:03:43 -0700 Message-Id: <20180921150351.20898-20-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler. This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 9 ++++++++ 2 files changed, 41 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 317fc59b512c..c04e68ef47da 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -237,6 +237,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + asm_volatile_goto("1: wrussd %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -1; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + WARN_ONCE(1, "write_user_shstk_32 used but not supported.\n"); + return -EFAULT; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{ + asm_volatile_goto("1: wrussq %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -1; +} +#endif /* CONFIG_X86_INTEL_CET */ + #define nop() asm volatile ("nop") diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7c3877a982f4..4d4ac57a4ba2 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1305,6 +1305,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, error_code |= X86_PF_USER; flags |= FAULT_FLAG_USER; } else { + /* + * WRUSS is a kernel instrcution and but writes + * to user shadow stack. When a fault occurs, + * both X86_PF_USER and X86_PF_SHSTK are set. + * Clear X86_PF_USER here. + */ + if ((error_code & (X86_PF_USER | X86_PF_SHSTK)) == + (X86_PF_USER | X86_PF_SHSTK)) + error_code &= ~X86_PF_USER; if (regs->flags & X86_EFLAGS_IF) local_irq_enable(); } From patchwork Fri Sep 21 15:03:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9174415A6 for ; Fri, 21 Sep 2018 15:10:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80E6A2E42F for ; Fri, 21 Sep 2018 15:10:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D8C42E47C; Fri, 21 Sep 2018 15:10:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6FFE62E430 for ; Fri, 21 Sep 2018 15:10:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 965308E0012; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 877D18E001A; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69F848E0012; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id D669C8E0012 for ; Fri, 21 Sep 2018 11:08:53 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id l65-v6so5753110pge.17 for ; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=UkpA98zALE0fynViTxAnI8knAwJHClX6QGtLN9Wtgik=; b=cvktly9M8r0mJ/qfAfXZ6FR6LKv91l4n5i5Evb+gqH1zvhN45/QC+LcosHcHCmnPg6 /eWU+Shk5jwIZViZvvHHrPU5FVyWcRua5Q4v4yiUdV8m5p9w50ZqA6bANHTPgx9odQDU vCl55rjaPT+PDTKRj39v97QvOSJLCYK0nQ/0aPV42c9ufgmBoRWODGr73/Ol6gEX1AwB BS6Tme/fwLG9YdzX/gPZdhwZdm/xK06b4613JVvT2NOk0XGZr+1nbxDHcAjWjvGOcU1a TEuJ5EYi5Sm2vm0Bk585asdIMX8pz9z61RqRBri5z9kPfTAnjkW3ZUVskcPT/h+CVeu+ RenQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51BB2FCGrNonMg0uEHQAKBGxQmARzLCc6ZLONcb6/btcm0S5TbIt I2wmL2kyjd6bGudklVhAQNP2DnFIVZ7kpHSA5qEQt48FFqQrPWTIRLL2ddFkOW9hjqKO/wnOAt/ 2mW59EZCpWnHKvlox1tUlE95qBK0cq26XmwDF4lVcj2CXwqGD8dYdEAX+K5AVqvz5sA== X-Received: by 2002:a17:902:1025:: with SMTP id b34-v6mr4198634pla.201.1537542533535; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdaf33UVX7D1geADFw2uJIWrpSNDCd8SEf6GDrSKNDZfgvYl3xns6VKsKrhkepaOULKTXQD7 X-Received: by 2002:a17:902:1025:: with SMTP id b34-v6mr4198548pla.201.1537542532155; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542532; cv=none; d=google.com; s=arc-20160816; b=GzEevFhX1iLWuvJHYD71BTq/605ru9hHFHwaKppGMBNRiz/uG/8T/Kt233Y9UJC+rS w7s2zp02AyQl7xXK4m+1oMoAmB08QiZPtGwvGX+Sdg3ihQFeA918itx35r7LVT5Bg2NW 9ZJK7T/Znf8rFSEcfV4svzwaJsXGAiIcRsPg/FIlL2PKMPZBiMk4ZhTNXpCm0oStMP6w ovelZyNkGbIjOqkNak7k0Jn+DyBdReQsfNwbLa2Xu61CsEHZSUdz7TyIo7QvHWlXgPyg U6qMijqcTXDmB2sDhwm1lXPJWqojUi3LaNaw3mApbkWLcK41tRBX6j3LTHk5lK57hQa3 XsVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=UkpA98zALE0fynViTxAnI8knAwJHClX6QGtLN9Wtgik=; b=AaE8GyKkPeQOmTX7EIcTcfamaW6QCQ0VwjLQRzQ5qNAWmUAnU7jeg/Cpsx7nwFc47M klt6IxF5RBbf7EZzdR/Tny2KtPwbnQPw+bPhRMeYgDmQQgFEq6Kkf2ch+gK2wywCBVhi QLU8WbdS+fh4Ujcc1uTJEuL3QP4UtD/BY20fR1NaVA0iTWXoaWFk8V286RzNm0YlZbfv 5tIY3S1/1OXsvzXhjzrHjFrmdN3He66orB5mtkDsBAQi4VYyaJcDNMEB+W4SJXHa0fUa UopdPVSQOtUHx6uL487eicHNVDJKk296blJCu4RoWBgPDb0SKL1XXOQobL24o2zk1oaw FhcQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r7-v6si26956600pga.77.2018.09.21.08.08.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856590" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:50 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 20/27] x86/cet/shstk: Signal handling for shadow stack Date: Fri, 21 Sep 2018 08:03:44 -0700 Message-Id: <20180921150351.20898-21-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer. Signed-off-by: Yu-cheng Yu --- arch/x86/ia32/ia32_signal.c | 13 +++ arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/asm/sighandling.h | 5 ++ arch/x86/include/uapi/asm/sigcontext.h | 17 ++++ arch/x86/kernel/cet.c | 115 +++++++++++++++++++++++++ arch/x86/kernel/signal.c | 96 +++++++++++++++++++++ 6 files changed, 251 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 86b1341cba9a..cea28d2a946e 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -34,6 +34,7 @@ #include #include #include +#include /* * Do a signal return; undo the signal stack. @@ -108,6 +109,9 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, 1); + if (!err) + err = restore_sigcontext_ext(buf); + force_iret(); return err; @@ -234,6 +238,10 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, if (fpu->initialized) { unsigned long fx_aligned, math_size; + /* sigcontext extension */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, @@ -277,6 +285,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, if (ia32_setup_sigcontext(&frame->sc, fpstate, regs, set->sig[0])) return -EFAULT; + if (setup_sigcontext_ext(ksig, fpstate)) + return -EFAULT; if (_COMPAT_NSIG_WORDS > 1) { if (__copy_to_user(frame->extramask, &set->sig[1], @@ -384,6 +394,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index ad278c520414..d9ae3d86cdd7 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -19,10 +19,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); +int cet_restore_signal(unsigned long ssp); +int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} +static inline int cet_restore_signal(unsigned long ssp) { return 0; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + unsigned long *new_ssp) { return 0; } #endif #endif /* __ASSEMBLY__ */ diff --git a/arch/x86/include/asm/sighandling.h b/arch/x86/include/asm/sighandling.h index bd26834724e5..23014b4082de 100644 --- a/arch/x86/include/asm/sighandling.h +++ b/arch/x86/include/asm/sighandling.h @@ -17,4 +17,9 @@ void signal_fault(struct pt_regs *regs, void __user *frame, char *where); int setup_sigcontext(struct sigcontext __user *sc, void __user *fpstate, struct pt_regs *regs, unsigned long mask); +#ifdef CONFIG_X86_64 +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fpu); +int restore_sigcontext_ext(void __user *fpu); +#endif + #endif /* _ASM_X86_SIGHANDLING_H */ diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/uapi/asm/sigcontext.h index 844d60eb1882..74f5ea5dcd24 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,23 @@ struct _xstate { /* New processor state extensions go here: */ }; +#ifdef __x86_64__ +/* + * Sigcontext extension (struct sc_ext) is located after + * sigcontext->fpstate. Because currently only the shadow + * stack pointer is saved there and the shadow stack depends + * on XSAVES, we can find sc_ext from sigcontext->fpstate. + * + * The 64-bit fpstate has a size of fpu_user_xstate_size, plus + * FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext + * is located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; +}; +#endif + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index ec256ae27a31..5cc4be6e0982 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -18,6 +18,7 @@ #include #include #include +#include static int set_shstk_ptr(unsigned long addr) { @@ -46,6 +47,69 @@ static unsigned long get_shstk_addr(void) return ptr; } +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +static int verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp = 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (ia32 && (token & 3) != 0) + return -EINVAL; + else if ((token & 3) != 1) + return -EINVAL; + + token &= ~(1UL); + + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + if ((ALIGN_DOWN(token, 8) - 8) != ssp) + return -EINVAL; + + *new_ssp = token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp = 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr = ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |= 1; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -107,3 +171,54 @@ void cet_disable_free_shstk(struct task_struct *tsk) tsk->thread.cet.shstk_enabled = 0; } + +int cet_restore_signal(unsigned long ssp) +{ + unsigned long new_ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + err = verify_rstor_token(in_ia32_syscall(), ssp, &new_ssp); + + if (err) + return err; + + return set_shstk_ptr(new_ssp); +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, + unsigned long *new_ssp) +{ + unsigned long ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + ssp = get_shstk_addr(); + err = create_rstor_token(ia32, ssp, new_ssp); + + if (err) + return err; + + if (ia32) { + ssp = *new_ssp - sizeof(u32); + err = write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp = *new_ssp - sizeof(u64); + err = write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + set_shstk_ptr(ssp); + return 0; +} diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 92a3b312a53c..e9a85689143f 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include +#include #define COPY(x) do { \ get_user_ex(regs->x, &sc->x); \ @@ -152,6 +153,10 @@ static int restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, IS_ENABLED(CONFIG_X86_32)); +#ifdef CONFIG_X86_64 + err |= restore_sigcontext_ext(buf); +#endif + force_iret(); return err; @@ -266,6 +271,11 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, } if (fpu->initialized) { +#ifdef CONFIG_X86_64 + /* sigcontext extension */ + if (boot_cpu_has(X86_FEATURE_SHSTK)) + sp -= sizeof(struct sc_ext) + 8; +#endif sp = fpu__alloc_mathframe(sp, IS_ENABLED(CONFIG_X86_32), &buf_fx, &math_size); *fpstate = (void __user *)sp; @@ -493,6 +503,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig, err |= setup_sigcontext(&frame->uc.uc_mcontext, fp, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fp); + if (err) return -EFAULT; @@ -576,6 +589,9 @@ static int x32_setup_rt_frame(struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; @@ -707,6 +723,86 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) } } +#ifdef CONFIG_X86_64 +static int copy_ext_from_user(struct sc_ext *ext, void __user *fpu) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (!access_ok(VERIFY_READ, p, sizeof(*ext))) + return -EFAULT; + + if (__copy_from_user(ext, p, sizeof(*ext))) + return -EFAULT; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + return 0; +} + +static int copy_ext_to_user(void __user *fpu, struct sc_ext *ext) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (!access_ok(VERIFY_WRITE, p, sizeof(*ext))) + return -EFAULT; + + if (__copy_to_user(p, ext, sizeof(*ext))) + return -EFAULT; + + return 0; +} + +int restore_sigcontext_ext(void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + + err = copy_ext_from_user(&ext, fp); + + if (!err) + err = cet_restore_signal(ext.ssp); + } + + return err; +} + +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fp) +{ + int err = 0; + + if (boot_cpu_has(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + unsigned long rstor; + + rstor = (unsigned long)ksig->ka.sa.sa_restorer; + err = cet_setup_signal(is_ia32_frame(ksig), rstor, &ext.ssp); + if (!err) { + ext.total_size = sizeof(ext); + err = copy_ext_to_user(fp, &ext); + } + } + + return err; +} +#endif + static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { From patchwork Fri Sep 21 15:03:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610269 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75D0A17EE for ; Fri, 21 Sep 2018 15:10:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6460A2E448 for ; Fri, 21 Sep 2018 15:10:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61E1A2E459; Fri, 21 Sep 2018 15:10:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CFBB2E4AA for ; Fri, 21 Sep 2018 15:10:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8CA38E001B; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CBB808E0019; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A21808E001B; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 4FBBF8E0019 for ; Fri, 21 Sep 2018 11:08:54 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id e124-v6so5756379pgc.11 for ; Fri, 21 Sep 2018 08:08:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=wdopx4IVSkT+toJhheCDf1jOyLb22h+VhiY1lHIGgTk=; b=YO6wOFginbXL3ZhADhXkd/NPMPX7mcrr5rkLKPoo+y1sD+tU260tscW6aeg67ipTIz r4vXjdYsAMSTJGY22ovFCUquiEfVcnp4CblvJ3r8l1A+bAZDU/DxT39Kq/q7/SMI+CAn ln15rbdjeKK0qBheXDphk66Y1LoPy4zVEA93dt6oFwk1DJdrakTkIVwP0H0ksrwfGQOn QsXryIXeQ5snYLA/NUAvjQGcBCyHuDsFbU7PrqHmSWmJF+ZhVWP0q9vsruJBEaZd9QaV qi7gR5wISzd/wF4J3wgjr1foCPxRLK2cvCh6EDSA+6HpTdlm7fbBbW4sayQX3IdzVFL6 rqxQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DemiMQiXhWwrmCNc2ktFkAt3dcyAQnFlouXTmaKBIprUIJ/IOV m4eVLwND+fwk6A6tdkuLMaJYbMvBxjtCh9B7yUyT6QuHImbEwJSff43NvSBjW1rDzJmlp359Sqc BEBOGAazeuNNG9hBGpKD5SQixHMJSc2wCCeA5D8Bfm/V3ZN0GT4C5ra67WsgwVcWZ4A== X-Received: by 2002:a17:902:b10b:: with SMTP id q11-v6mr9999176plr.90.1537542533963; Fri, 21 Sep 2018 08:08:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYAJ9+/JJOrbAb3wBOY0DIiRMMbO//tb85vyEUr376ylmCUHRtlLAVUsvAjjN9QcNJp1Xqg X-Received: by 2002:a17:902:b10b:: with SMTP id q11-v6mr9999085plr.90.1537542532496; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542532; cv=none; d=google.com; s=arc-20160816; b=aQy/UF18TgRwdUeLXSY+GtUrqV9dPTtrWdE+JCgkX/VeTPFSdVNFa8pW/4hx+F7EZH GocZfYv7s01RHl7wl028cit+tmX0Uwp4PYW+yOC3Pff+fcxHVmipltIeDAdPs9gbTKOx DNVOxmphbI9wnSwJvU3jVdRLH8ukLVxSz1ztYqakdHUqtryz2rNIB/P5ZIx9y6p6QxMS 6wsomwfYOt0HxLANMpXB349lNKzpnjBpD/p5nszEcEuM28soHqoxlE5R8SQTIv3S9MGv 0QAf2yF/krj60qkQvDde+SN+X6OiT6EENb3lFzDSzMmUBs5ksCAPf1T3/eabNnm4nHwV nDYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=wdopx4IVSkT+toJhheCDf1jOyLb22h+VhiY1lHIGgTk=; b=wNBKuUbuSTeHjQAsxowqf9wuGd5KxpCCh+ZLRLc1LDDhf1Lh+auRmfYjofRxNnLUeC rBgtOILXE4DziUVxkenenPWgAupb1babkT009nR3lGO8RwWNVqpXjhHl5JAwfpdO0f9B jfANLqNlja1708L3otIRho5MZF0rrGBXO89390u20tRrUHGEQwvow+M+s9z+JhpABbqR ZdTmNvpUCBhhE5wBlFirMYvUlSO1/9pbqjanUng8Q4hApQklOgiWIuKtGHGxJsAlyC/5 lgy09CAOwHRF7tjWiw7i1lQsTcpukUCgfJiuOytjweuk5NLTYRSC62Mhe8QE1/dnWb0S L66A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id d11-v6si26378966pgh.564.2018.09.21.08.08.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856593" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:51 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 21/27] x86/cet/shstk: ELF header parsing of Shadow Stack Date: Fri, 21 Sep 2018 08:03:45 -0700 Message-Id: <20180921150351.20898-22-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 4 + arch/x86/include/asm/elf.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/elf.c | 340 +++++++++++++++++++++++ fs/binfmt_elf.c | 15 + include/uapi/linux/elf.h | 1 + 7 files changed, 382 insertions(+) create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/elf.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 808aa3aecf3c..6377125543cc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1919,12 +1919,16 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_PROGRAM_PROPERTIES + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n depends on CPU_SUP_INTEL && X86_64 select X86_INTEL_CET select ARCH_HAS_SHSTK + select ARCH_HAS_PROGRAM_PROPERTIES ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 0d157d2a1e2a..5b5f169c5c07 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -382,4 +382,9 @@ struct va_alignment { extern struct va_alignment va_align; extern unsigned long align_vdso_addr(unsigned long); + +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +extern int arch_setup_features(void *ehdr, void *phdr, struct file *file, + bool interp); +#endif #endif /* _ASM_X86_ELF_H */ diff --git a/arch/x86/include/uapi/asm/elf_property.h b/arch/x86/include/uapi/asm/elf_property.h new file mode 100644 index 000000000000..af361207718c --- /dev/null +++ b/arch/x86/include/uapi/asm/elf_property.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _UAPI_ASM_X86_ELF_PROPERTY_H +#define _UAPI_ASM_X86_ELF_PROPERTY_H + +/* + * pr_type + */ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* + * Bits for GNU_PROPERTY_X86_FEATURE_1_AND + */ +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + +#endif /* _UAPI_ASM_X86_ELF_PROPERTY_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index fbb2d91fb756..36b14ef410c8 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/elf.c b/arch/x86/kernel/elf.c new file mode 100644 index 000000000000..2fddd0bc545b --- /dev/null +++ b/arch/x86/kernel/elf.c @@ -0,0 +1,340 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Look at an ELF file's .note.gnu.property and determine if the file + * supports shadow stack and/or indirect branch tracking. + * The path from the ELF header to the note section is the following: + * elfhdr->elf_phdr->elf_note->property[]. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The .note.gnu.property layout: + * + * struct elf_note { + * u32 n_namesz; --> sizeof(n_name[]); always (4) + * u32 n_ndescsz;--> sizeof(property[]) + * u32 n_type; --> always NT_GNU_PROPERTY_TYPE_0 + * }; + * char n_name[4]; --> always 'GNU\0' + * + * struct { + * struct property_x86 { + * u32 pr_type; + * u32 pr_datasz; + * }; + * u8 pr_data[pr_datasz]; + * }[]; + */ + +#define BUF_SIZE (PAGE_SIZE / 4) + +struct property_x86 { + u32 pr_type; + u32 pr_datasz; +}; + +typedef bool (test_fn)(void *buf, u32 *arg); +typedef void *(next_fn)(void *buf, u32 *arg); + +static inline bool test_note_type_0(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + + return ((n->n_namesz == 4) && (memcmp(n + 1, "GNU", 4) == 0) && + (n->n_type == NT_GNU_PROPERTY_TYPE_0)); +} + +static inline void *next_note(void *buf, u32 *arg) +{ + struct elf_note *n = buf; + u32 align = *arg; + int size; + + size = round_up(sizeof(*n) + n->n_namesz, align); + size = round_up(size + n->n_descsz, align); + + if (buf + size < buf) + return NULL; + else + return (buf + size); +} + +static inline bool test_property_x86(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if (pr->pr_type > max_type) + *arg = pr->pr_type; + + return (pr->pr_type == GNU_PROPERTY_X86_FEATURE_1_AND); +} + +static inline void *next_property(void *buf, u32 *arg) +{ + struct property_x86 *pr = buf; + u32 max_type = *arg; + + if ((buf + sizeof(*pr) + pr->pr_datasz < buf) || + (pr->pr_type > GNU_PROPERTY_X86_FEATURE_1_AND) || + (pr->pr_type > max_type)) + return NULL; + else + return (buf + sizeof(*pr) + pr->pr_datasz); +} + +/* + * Scan 'buf' for a pattern; return true if found. + * *pos is the distance from the beginning of buf to where + * the searched item or the next item is located. + */ +static int scan(u8 *buf, u32 buf_size, int item_size, + test_fn test, next_fn next, u32 *arg, u32 *pos) +{ + int found = 0; + u8 *p, *max; + + max = buf + buf_size; + if (max < buf) + return 0; + + p = buf; + + while ((p + item_size < max) && (p + item_size > buf)) { + if (test(p, arg)) { + found = 1; + break; + } + + p = next(p, arg); + } + + *pos = (p + item_size <= buf) ? 0 : (u32)(p - buf); + return found; +} + +/* + * Search a NT_GNU_PROPERTY_TYPE_0 for GNU_PROPERTY_X86_FEATURE_1_AND. + */ +static int find_feature_x86(struct file *file, unsigned long desc_size, + loff_t file_offset, u8 *buf, u32 *feature) +{ + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + u32 last_pr = 0; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < desc_size; done += buf_pos) { + read_size = desc_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) + return (ret < 0) ? ret : -EIO; + + ret = 0; + found = scan(buf, read_size, sizeof(struct property_x86), + test_property_x86, next_property, + &last_pr, &buf_pos); + + if ((!buf_pos) || found) + break; + + file_offset += buf_pos - read_size; + } + + if (found) { + struct property_x86 *pr = + (struct property_x86 *)(buf + buf_pos); + + if (pr->pr_datasz == 4) { + u32 *max = (u32 *)(buf + read_size); + u32 *data = (u32 *)((u8 *)pr + sizeof(*pr)); + + if (data + 1 <= max) { + *feature = *data; + } else { + file_offset += buf_pos - read_size; + file_offset += sizeof(*pr); + ret = kernel_read(file, feature, 4, + &file_offset); + } + } + } + + return ret; +} + +/* + * Search a PT_NOTE segment for the first NT_GNU_PROPERTY_TYPE_0. + */ +static int find_note_type_0(struct file *file, unsigned long note_size, + loff_t file_offset, u32 align, u32 *feature) +{ + u8 *buf; + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + + buf = kmalloc(BUF_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < note_size; done += buf_pos) { + read_size = note_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) { + ret = (ret < 0) ? ret : -EIO; + kfree(buf); + return ret; + } + + /* + * item_size = sizeof(struct elf_note) + elf_note.n_namesz. + * n_namesz is 4 for the note type we look for. + */ + ret = 0; + found += scan(buf, read_size, sizeof(struct elf_note) + 4, + test_note_type_0, next_note, + &align, &buf_pos); + + file_offset += buf_pos - read_size; + + if (found == 1) { + struct elf_note *n = + (struct elf_note *)(buf + buf_pos); + u32 start = round_up(sizeof(*n) + n->n_namesz, align); + u32 total = round_up(start + n->n_descsz, align); + + ret = find_feature_x86(file, n->n_descsz, + file_offset + start, + buf, feature); + file_offset += total; + buf_pos += total; + } else if (!buf_pos) { + *feature = 0; + break; + } + } + + kfree(buf); + return ret; +} + +#ifdef CONFIG_COMPAT +static int check_notes_32(struct file *file, struct elf32_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 4)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +#ifdef CONFIG_X86_64 +static int check_notes_64(struct file *file, struct elf64_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 8)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +int arch_setup_features(void *ehdr_p, void *phdr_p, + struct file *file, bool interp) +{ + int err = 0; + u32 feature = 0; + + struct elf64_hdr *ehdr64 = ehdr_p; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return 0; + + if (ehdr64->e_ident[EI_CLASS] == ELFCLASS64) { + struct elf64_phdr *phdr64 = phdr_p; + + err = check_notes_64(file, phdr64, ehdr64->e_phnum, + &feature); + if (err < 0) + goto out; + } else { +#ifdef CONFIG_COMPAT + struct elf32_hdr *ehdr32 = ehdr_p; + + if (ehdr32->e_ident[EI_CLASS] == ELFCLASS32) { + struct elf32_phdr *phdr32 = phdr_p; + + err = check_notes_32(file, phdr32, ehdr32->e_phnum, + &feature); + if (err < 0) + goto out; + } +#endif + } + + memset(¤t->thread.cet, 0, sizeof(struct cet_status)); + + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) { + if (feature & GNU_PROPERTY_X86_FEATURE_1_SHSTK) { + err = cet_setup_shstk(); + if (err < 0) + goto out; + } + } + +out: + return err; +} diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index efae2fb0930a..b891aa292b46 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1081,6 +1081,21 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES + if (interpreter) { + retval = arch_setup_features(&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + } else { + retval = arch_setup_features(&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + + if (retval < 0) + goto out_free_dentry; +#endif + if (elf_interpreter) { unsigned long interp_map_addr = 0; diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index c5358e0ae7c5..5ef25a565e88 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -372,6 +372,7 @@ typedef struct elf64_shdr { #define NT_PRFPREG 2 #define NT_PRPSINFO 3 #define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5 #define NT_AUXV 6 /* * Note to userspace developers: size of NT_SIGINFO note may increase From patchwork Fri Sep 21 15:03:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4266817EE for ; Fri, 21 Sep 2018 15:10:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 336CC2E451 for ; Fri, 21 Sep 2018 15:10:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 315EB2E4CB; Fri, 21 Sep 2018 15:10:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95F1B2E4BE for ; Fri, 21 Sep 2018 15:10:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25B6D8E001A; Fri, 21 Sep 2018 11:10:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 209BB8E0019; Fri, 21 Sep 2018 11:10:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F9818E001A; Fri, 21 Sep 2018 11:10:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id C30C78E0019 for ; Fri, 21 Sep 2018 11:10:17 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id d10-v6so6251119pll.22 for ; Fri, 21 Sep 2018 08:10:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=B3xJgz4x741BDQNBDqqpysPELaB2fR/elxB8Y4YSfZQ=; b=J6eBLMAitrIjA/WiUqFT1sxCQx0AFxQDmP9wDi4S9xO4QZAUyKx4xKjxLUreZyDlk9 ovJcPPbLQJNBo3E6AhPgyMAaouuJCzR+lQghHqgIh19RXJLDu1l7Wsj4XMYnv8s20DlH GlQQGE3mJWqhh/8QhEjWfWl8cf90zUBKT1+a2JhGB0nVyRxs6K8SACeEcJAuZMT39/+y SkS1i9+wCDibwTLonl4VBLo6aINs9+rng23JDeFV5z4iISLBrryHdU34Kj7iq1yZu58z x21L3kCfvRnWPrTSn2x+cxreDYjBRjduJjxFbf3GwpfKS5oQ2NerOis1tWsFD2xJoz/x AmPQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51COD6DjhHXyyDjp6bXQg5J3vSa65wQAmWxAGs4uZXtzGhjTVB3C qDlcA8UExHBgoeWZ92NTPpXZDkKzVTKaLqdYQ8GqENVm9c5An9Fh0t8SJbf2VeSp08goyBoBi8J cslRWGVcJ5Ak9g2ok24d7T3B819HCAr9D2v2HxvyvcuUIguNEsTQp6akIPXNiLVDwSg== X-Received: by 2002:a17:902:1001:: with SMTP id b1-v6mr45006133pla.155.1537542617469; Fri, 21 Sep 2018 08:10:17 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbFad84ofLUg2UILfMdHBMi03kojZb37+ukNDOHM8RKkr3/d6kfNrCYxVQPyj2DNaJb+5ti X-Received: by 2002:a17:902:1001:: with SMTP id b1-v6mr45000555pla.155.1537542532469; Fri, 21 Sep 2018 08:08:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542532; cv=none; d=google.com; s=arc-20160816; b=Xz7qNkamLvTPBqJ5bHmHX7laAs9RIac64cY2yLltGqNjf2vdrN19vZcZNpWK+FXvBq 4BxOeJX/ViRE9enTYLLgCqEGNMCA8qZGuQDKcLfl4+0nJN0gIbluxn8ExZ6U/zCqTcKw n7HFy2qZQ71rF/QhlDRTtsUkM6g68RqVoNcxqIQCBgoXkmosM/bSPadaVjFRfXB48lYc aVoPIwqEFirmzFCbsB55yznYSlD/3CLz9TJk9iqjLaP9rjEGSSDo80g/Ij1OQ48epD9b Tlage4Zw/jF4RcwK6UEzDwqdMqSd9KJHPEjmWtVEXAiQb1sFkUK5EiYYENatsDSho5Ln C28g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=B3xJgz4x741BDQNBDqqpysPELaB2fR/elxB8Y4YSfZQ=; b=0D3sRy/yTy6ves/8smq/yNwCg1qr1awvR7kOD9AZ567YICg9xxOBlneIYx1gDIl0Qh GAgQOw+ytH0JPDB+Smwz+c5TBFbdanw+kQBBsswcTXLSIyVifN8gz7OtmxgN5sVIYogW f8QAJvY5mCyV1kme20SfgxoKx03L0BQdm4RtvO+3dV11ePfw6QaW2KTME33RxVozEKIo A4bh8HgQxuZbiFDLfabWeD9yylqS8uZt0ggV7CJL0yVI/XF2uukw7QNlr8/Zgai+AznW jRmjSMIiDZawm38htWBLgTFyRQGGXe6wKLppJUFgDfSQ6M9LnJywqAAx3xL/pLj2qFHF y0cA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r7-v6si26956600pga.77.2018.09.21.08.08.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:08:52 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:08:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856598" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:08:51 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 22/27] x86/cet/shstk: Handle thread shadow stack Date: Fri, 21 Sep 2018 08:03:46 -0700 Message-Id: <20180921150351.20898-23-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The shadow stack for clone/fork is handled as the following: (1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child. It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK. (2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK. (3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret. This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 34 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 ++++++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index d9ae3d86cdd7..b7b33e1026bb 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -17,12 +17,14 @@ struct cet_status { #ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk(void); +int cet_setup_thread_shstk(struct task_struct *p); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return 0; } +static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return 0; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index eeeb9289c764..8da7c999b7ee 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -223,6 +224,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 5cc4be6e0982..ce0b3b7b1160 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -134,6 +134,40 @@ int cet_setup_shstk(void) return 0; } +int cet_setup_thread_shstk(struct task_struct *tsk) +{ + unsigned long addr, size; + struct cet_user_state *state; + + if (!current->thread.cet.shstk_enabled) + return 0; + + state = get_xsave_addr(&tsk->thread.fpu.state.xsave, + XFEATURE_MASK_SHSTK_USER); + + if (!state) + return -EINVAL; + + size = tsk->thread.cet.shstk_size; + if (size == 0) + size = rlimit(RLIMIT_STACK); + + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + if (addr >= TASK_SIZE_MAX) { + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + tsk->thread.cet.shstk_enabled = 0; + return -ENOMEM; + } + + state->user_ssp = (u64)(addr + size - sizeof(u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; + return 0; +} + void cet_disable_shstk(void) { u64 r; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 4a776da4c28c..440f012ef925 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -125,6 +125,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + cet_disable_free_shstk(tsk); fpu__drop(fpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index ea5ea850348d..9cdbd87bb908 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -325,6 +325,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp, if (sp) childregs->sp = sp; + /* Allocate a new shadow stack for pthread */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM) { + err = cet_setup_thread_shstk(p); + if (err) + goto out; + } + err = -ENOMEM; if (unlikely(test_tsk_thread_flag(me, TIF_IO_BITMAP))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr, From patchwork Fri Sep 21 15:03:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610279 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7FF7F15A6 for ; Fri, 21 Sep 2018 15:10:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 719222E4E8 for ; Fri, 21 Sep 2018 15:10:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F28C2E48A; Fri, 21 Sep 2018 15:10:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 13EF72E4C4 for ; Fri, 21 Sep 2018 15:10:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F05758E001D; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E8E3B8E0019; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA3418E001D; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 9C4EA8E0019 for ; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id 191-v6so5738536pgb.23 for ; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+hAwkPRrWa1SdzA9lzXBTpLorf0142oXjZbO3utwlJg=; b=Zg62O24vKhph0/wxNzb3yK5vR51BagttjTbsCqmEp1QJvMSuVJsej4Az8bQ40Rcc4p V0r0hNdC7h+pyspGDQyUazLlumgjE9phSIaFrK0Yj6YwcqC56Wc8gKfG0Hl5IhqUMB9F PyqFvTxd/H0D/JMqIj8zxzlJQiBgudGwqxjZOo+WZ3//gEk4ZDMCFC+ODvGbXo+VESYW 7QkQE5DzGr5jQsX5bnYHGIRTwFrB8SKAKFrejtZLfdbHKMVN2SNAb+FhBhNG8pIW7q5I Oodfyp3xZMFm6yAoSB9ps3O5YWoeHpXyRa/30UpxUIN9NbSJVKaFluGf37IuXa36yaeB Z3jQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CJEHajUubeIg6Nsx8ltPEz+p9uTK6XWzUdqPRyWDYizZdlViN9 i4x9LXpyj9ZYoXLwz2UIFW124Lm0YJMb2tzRMriB7aigF6nZrbmHL2Ezo4KL5g8AcfLEqDmG4DR 6uHlCvrArxF/KYiFxYTIx6jdffoDBPY9Z/UDOqhAB0DgKssh4Ds6kKUHEGFr5v4sf5A== X-Received: by 2002:a62:54c7:: with SMTP id i190-v6mr46774997pfb.155.1537542626342; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZT12X9DienIUFQRV9sX1d/aXdeWhGX/YVp29qgPkGhCN+QAmiaFq38msTLq0+VTMAJOhyj X-Received: by 2002:a62:54c7:: with SMTP id i190-v6mr46774928pfb.155.1537542625436; Fri, 21 Sep 2018 08:10:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542625; cv=none; d=google.com; s=arc-20160816; b=cYN7oteaih2HPvFtKR8ZAVqPW0nhibYTHs5yYOx3Vm/yf3dlzATWHl5SstqKjBdcb9 1723xfBZA5kuvHY58qr+tzKmt8zTsw04wE1nypAToclUvCd6RYIQ2efGneRua3lRW2vl x5IGV0jOYcTKKaXOJX2CgYIs9IRhbCgbiDA9hcvq45LUJ7QsTn/mOWmrOfDhlCSgJtyt E0X84+hhhOGVwVaHW5bOUxr5JpiOMWpHCPhKXycYLm9qDB9JIfnamCthN1OemUA0xjG1 WwSL7NoZAaZKU3B/LCNbRJK8wWj0cIbG2/tPVLuK/74gHsX6m3byuU9HAP69wF6LWbkS 1r9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+hAwkPRrWa1SdzA9lzXBTpLorf0142oXjZbO3utwlJg=; b=PqcR3dYgj6tuIe6jnHPLoHGV9QQ0E0bJu8Y1CDmeHsABXUjZmeQlkg+yVJwHEXo9M1 +WcKZmG/D/Qqre7fQINkX+Mb1zVNDoxgrD72jAINhVzR7jk4KyW/5/Sq9CdqitPeOVpx u6rKr0a4QxLhwbEowxXwufOFppMGytLZBJjEDOEL7IinPk9QTV2aqsH/EtdmJqj7QTam Tx9bbQH6OJAFmQ6lqwy5XLbDLmlWcwTCx5icEblxaumaokSEBTgWSNw1punFLF9/fjyO +xZnTndANYExTPJZOJuxqu7Dq+cfFnXD1hNaDxEHz8M7MJpi87f7yh6PtYFGg0K379qM tmcA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id z3-v6si26822954pgh.557.2018.09.21.08.10.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:10:25 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:10:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856865" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:09:53 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 23/27] mm/map: Add Shadow stack pages to memory accounting Date: Fri, 21 Sep 2018 08:03:47 -0700 Message-Id: <20180921150351.20898-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add shadow stack pages to memory accounting. Also check if the system has enough memory before enabling CET. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 5f2b2b184c60..de2d0faa1c61 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1671,6 +1671,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) if (file && is_file_hugepages(file)) return 0; + if (arch_copy_pte_mapping(vm_flags)) + return 1; + return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE; } @@ -3261,6 +3264,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; + else if (arch_copy_pte_mapping(flags)) + mm->data_vm += npages; } static vm_fault_t special_mapping_fault(struct vm_fault *vmf); From patchwork Fri Sep 21 15:03:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A06F15E8 for ; Fri, 21 Sep 2018 15:10:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0AA212E471 for ; Fri, 21 Sep 2018 15:10:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 074702E501; Fri, 21 Sep 2018 15:10:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3E442E4FF for ; Fri, 21 Sep 2018 15:10:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E695D8E001E; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E189E8E0019; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF71E8E001E; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 764098E0019 for ; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id 186-v6so5746418pgc.12 for ; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=coov9TMBwRb+m8/Fllbc10SqeFfZ/zqjB+YRUB70eiI=; b=rbljXv7IAh+9AF8mReVwTtSoku6RV4VuDXuxqCRm3lHEER5eDAzXTzvtjQioOqaYVP gMNR9v0bVKLS2+AGeYn96JDPRFZy1LATMP58DngO6u1nBc+40l1FG0GoheXeK0+IClGL HDgtfRWLniuXSfGktKOxJfKGaqFlyoNNSIvm2t2EXMU115wZ6rZ3tSRkAlRYW8SskA2F SGc1RP4Wi5i3P2bdrXhdhkjuxhDDbpkznWXBoFLYJ0eEmjf9YnbWpVmgS98rOoNpfhxJ F6v0BCu6dSjLxr8Ef6R5HjuKh1Tyeff+l6hcHZrRM19oj3mAcOnLB6filcQS253Yibp9 z7Qg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51AXYB2KvBoBsd3fkBZYKbj1kD7X3/uOHK2MU1Awe99BDffASB7e oUZVAqLJF+1IaoZPueD7gsFqy5GTrmQhsleN6ecsE2Ut4YvOe8ImbAmFhN+bkkgu54BwNWvK+Ua /jURPO596h0VBMNiQzXc2QHhCBsGDxLRiyOCGgBea536RyhSn6fwyuwt9sqThRTYLTQ== X-Received: by 2002:a17:902:47c2:: with SMTP id d2-v6mr45446700plh.317.1537542627190; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-Smtp-Source: ANB0VdatNZJCwIWNPgdzod1jXkUbNrEIp23eRIQ5jN0XFnE/nkbzVWla/JzQnVkwEthmP80fXaMr X-Received: by 2002:a17:902:47c2:: with SMTP id d2-v6mr45446641plh.317.1537542626268; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542626; cv=none; d=google.com; s=arc-20160816; b=QI/04jo+K/oGvBLjtx7qgGmHtO214z2VkXGuVLY8pr+3x1r7+Rk1NFwIAi26SS8ITy IeMkq/RwxBDcrMrERgsihIbBCgexg0mGFVhA6+OnGFX4x0AP+Pfb9r3jdr8Ux4Eipsz/ Ezjv5vURLzHm0ousq3HRw1Sy4ss5iBd4tLGmDaG1IjLh9nmBwIEmODaaefBwG8yE/AFC ioUuPN/008WdaG5xCUWPM07h7sClrdL2L3MND5oD95sjomdMmL5osD8aFp8/NpA6hdeM MCJcob8L+tSmMiCqm3U6ATRcC4nECf2WzqxF+95pWQGIwoMV6CJjaRNPR5jMj+rMv8xS i0vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=coov9TMBwRb+m8/Fllbc10SqeFfZ/zqjB+YRUB70eiI=; b=DfW9yKRDs07jrxvzDeegBTg7GMlBNjlUu0ql2Cm+SNJWxJ8vIRLkIbPGadmc7wX0CB zNrlzcjZTR31hQ5gns8zvZZOJCd5wLvtxXtOz/EluIcJpsUyl211z3buf6To0WRwGKlv SLXAycM7aSmcV6tzhrmkcNOs6JajRoGMOrCd4d0knT0mFSZxOzdl4TqHKJhc3Ie7Mw5v 41VxIUdhIntxSEpy2hsFAoZt2X1jxizTymR32Hm0yttMvWKsqVw0PlEVh1fpQ1V54+DH tPL1R+ow+NiCaqOTgxUBp+4hByk5WHkCv45mXZKxk4SAiB2hntS2xQz7QIYHqTBTq+Md EybA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id z3-v6si26822954pgh.557.2018.09.21.08.10.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:10:26 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:10:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856869" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:09:54 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 24/27] mm/mmap: Create a guard area between VMAs Date: Fri, 21 Sep 2018 08:03:48 -0700 Message-Id: <20180921150351.20898-25-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Create a guard area between VMAs, to detect memory corruption. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c4cc07baccda..3a823bdae09d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2443,24 +2443,34 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m static inline unsigned long vm_start_gap(struct vm_area_struct *vma) { unsigned long vm_start = vma->vm_start; + unsigned long gap; + + if (vma->vm_flags & VM_GROWSDOWN) + gap = stack_guard_gap; + else + gap = PAGE_SIZE; + + vm_start -= gap; + if (vm_start > vma->vm_start) + vm_start = 0; - if (vma->vm_flags & VM_GROWSDOWN) { - vm_start -= stack_guard_gap; - if (vm_start > vma->vm_start) - vm_start = 0; - } return vm_start; } static inline unsigned long vm_end_gap(struct vm_area_struct *vma) { unsigned long vm_end = vma->vm_end; + unsigned long gap; + + if (vma->vm_flags & VM_GROWSUP) + gap = stack_guard_gap; + else + gap = PAGE_SIZE; + + vm_end += gap; + if (vm_end < vma->vm_end) + vm_end = -PAGE_SIZE; - if (vma->vm_flags & VM_GROWSUP) { - vm_end += stack_guard_gap; - if (vm_end < vma->vm_end) - vm_end = -PAGE_SIZE; - } return vm_end; } From patchwork Fri Sep 21 15:03:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FC4415A6 for ; Fri, 21 Sep 2018 15:10:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FBCF2E460 for ; Fri, 21 Sep 2018 15:10:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3D39C2E4BE; Fri, 21 Sep 2018 15:10:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2945C2E460 for ; Fri, 21 Sep 2018 15:10:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A8C08E001C; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2591D8E0019; Fri, 21 Sep 2018 11:10:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FD408E001E; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id B8F888E001C for ; Fri, 21 Sep 2018 11:10:26 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id z18-v6so6712210pfe.19 for ; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=AOHK0mK6OMLhyVWfypXS6nFx+0YXj0Cflh1jDhzHtRg=; b=oHF7zc/M3UEj4rcWCOC2rB3eY5jaIVbGeMdlyz0wExrULv2E+lDcRxbpUJlvRSxquW g+/RiX4uI6ZslJVfsLvMNowwub5LXSB5XCe9QUy+alkz03QZqyLRtuPTdpbtXaD8Sw4q +KN7W8y+AEW3wbGaRg8wrWCATPwCrwg98DVQhJWM3bNz3JRdzsg76CdN9YPZ9esas/lQ gGxqdTTGFXJJjU1BhYntcwfczJfogJdKlnE2jC4mxQLcQaJtCZiX2HYifRviASboCWOx +FBWxsftyTiXCldihyEh6wrlHRwRp6TvPFwX57qmBULTx43ex4/isS15IhxEWMQVShGm SQMQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51B0JR6Z2pt6di1tvXD98R0KNL6ZBKEtLsXYuoJQlyhWPrrUrJf4 FCQarhhC0eJSYVE98LJNVOGOkPrL5//Hsbcr5k1VQ+gXCzads/kD/zE4tDMvs7H+cZ2z5kP7nPJ zIvvykRe4ZmYiuI5FG2Bsa+nZngXWgtXFgIMnJ4fiTO4YXpmN47gqITCCgaiBZ5gszw== X-Received: by 2002:a62:4494:: with SMTP id m20-v6mr47393411pfi.205.1537542626459; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYJFWSaBVOGFQl5nT2NsUCzLxjfUoWTygaHBba5ybPAR/rdlTMCYnFHTZI7cR7tlFghOdjF X-Received: by 2002:a62:4494:: with SMTP id m20-v6mr47393342pfi.205.1537542625704; Fri, 21 Sep 2018 08:10:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542625; cv=none; d=google.com; s=arc-20160816; b=bsIkYiqlaqxkoO/94lm2mJ6sqrsYjWT7uQvBdRWalUzBtv+e3YbTidK2+Joqbpemhb QL+Ei/q7/FPl8+FnL0MRjkqTDQgb6R587nsAutLjqK6dlvnIpEwFBwldJlae8NZNMA0a 9sTAHVRWbi5IkE4YCZn2EToqlIzh9B5FyVca1bRmUODoT8khaluCAiavJEzn2mIErqP3 +bDJsXnqekNa/5vyM2Ial7oy3mncOmPhqYfKlHScKqOAASJSypQlBhcjIw4ugr7co5E1 I8qESHwj6t1dF9A/wTpeWwwY1PZmuEDHcjA+30ZdD9wnDZAgN8PAH7WlUSTB2m9j/4k3 kiRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=AOHK0mK6OMLhyVWfypXS6nFx+0YXj0Cflh1jDhzHtRg=; b=0A+uNqUfaysmQcaS7rNrUGnRBqZAdXDluBmzV0Z85X/jukoFiF55uMPP7WA2r5K1F1 u3EquRpVcnMxTvEaiax677su+DkK09k7Rjc2s34drIytTN26IJ0JRHwcaPzI+I7cMA/F kPc5gxHsIKqN6608nvuLHEfVfl7KQ8lS9PGhXlMbJscZieSvGeA/PIAd9JBm7dgKYFkd i50KWP+7AeQtb89qL+ql9kt6S2pQnJOwyRgUgrG6d90jNdWRQ0MwV4ljeeCq8BEtJE+J 350F4mDZEh/yFd7fMSJ8BebcWVyN4IethOcBxawybrkDOdlnTsJbWhbFvGNq5fNrS8+q d1VA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id z3-v6si26822954pgh.557.2018.09.21.08.10.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:10:25 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:10:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71856872" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:09:54 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 25/27] mm/mmap: Prevent Shadow Stack VMA merges Date: Fri, 21 Sep 2018 08:03:49 -0700 Message-Id: <20180921150351.20898-26-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Function returns could unwind stacks beyond its allocated area. We do not merge shadow stack areas. This and VMA guards prevent shadow stack underflow. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index de2d0faa1c61..fa581ced3f56 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1123,6 +1123,12 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + /* + * Do not merge shadow stack areas. + */ + if (vm_flags & VM_SHSTK) + return NULL; + if (prev) next = prev->vm_next; else From patchwork Fri Sep 21 15:03:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9496F14BD for ; Fri, 21 Sep 2018 15:10:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 83ED02E4C4 for ; Fri, 21 Sep 2018 15:10:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8194A2E524; Fri, 21 Sep 2018 15:10:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB0E42E4FF for ; Fri, 21 Sep 2018 15:10:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B942F8E001F; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AC0018E0020; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FF898E001F; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 2E5B38E0019 for ; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id n17-v6so6668351pff.17 for ; Fri, 21 Sep 2018 08:10:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=qpWjjwS69NtRuZcBDki/uhmcm+XjXpFLfgHPaAMpcEU=; b=HBeVXgQgzaxjPHDx5Q10g99UUs3liPQnu6Z9OvkVIyusJrcvD2lwAmrZB5VSYYdUrn 6Hlt54LHBAHqJ15hRi5nTvCOILHfNT9i7o5W8Y7OtMjk2GgfBk5MduHk5VtG4Sh6tu90 z8jbuwchEx1K666ILhGp7M1LRtzkuayC6g64ccVeDaKpsu/gJ1dV8OC1BrpZP03CXmsH qdLzZWnAivegIoi18fnd6trwK1qXm8w7xNToR3KsUcfT6uOMXF7ZjpOjjvRvqnextRyp YtgwwkBMWjDdCSAZ+6vamL0okWctT8+SxtkvWk+wZguEwkpakZlRMxq5L8d4qPJfmCXJ le6Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51D0IOU9+GEotYtAHFH8/EHYOgUXwYlG7XVZrgo6Ha2tM9vFivj5 TLo5bHstO78PDyuD0cdEYBXnLCKWYfhiM6ZbL0/C5tEzYuEj6MeVWlrt9TZNSAI0NoQJ9vdJ759 TVn5S6aw51HSeNSzLcLWviDuKxYYPxzx+hFVqUUAwRSWgh+VNxQrB94rKmjYKEQSf7Q== X-Received: by 2002:a63:fa49:: with SMTP id g9-v6mr30047449pgk.18.1537542627809; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbyNsMV42F7qYRcAWjpwKd5hbYBd1hd5x3gfkqGRHSbQrOrEF1Hht+NBexJaFWRtOpdHNV2 X-Received: by 2002:a63:fa49:: with SMTP id g9-v6mr30047356pgk.18.1537542626573; Fri, 21 Sep 2018 08:10:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542626; cv=none; d=google.com; s=arc-20160816; b=CJncJBwK818nD0A2QC8Zow0zsDlduGO52Qyiyo7m6blXVe4IPcFXpSI7qmvN2F3kJO TtG3Lb046+MClz8PYHcNi/VvH/AL/isoBRlixMQM7wVSae7jVqN8wMLnPkYHRXpf30sJ 52jxgvSkvIlefibA0npqOp7OsY0GuK2MllBYJ+02Vl8ejAi4quNabhzVcRrIz8nbbKWs DsGnCQOJnlkymnMP58lW/N53AiVWGTDVfa4l/ZmY6XDfmFoyIw6UmFoCFWO+rUM3A5v0 hQCKFTg0zhVAgA2yNSkUjlH+cdKpKh5Bk2vtqPwz0NaMtbPPnd6hfNY0J+KBC5TqUt8B ftZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=qpWjjwS69NtRuZcBDki/uhmcm+XjXpFLfgHPaAMpcEU=; b=FhPoB990uUL6GExegUbpKwtWhVDzfEift4Jidy/1tnMCQZhxSedRaUIukXbId1lqIk 7+gcnDAJbDNWRZyTUaPejcyKbcLwTCgSG+5fTnZBbaNeYgW76J/iCE3wzZOQ7Xd09ujk R8rYHOALxLuLDOU2ZF8AvjUcit/2+XDy0XiACoePBTDs2Sun7mTcvR4pyY26L137IBGx N7flU6EFwWEB/F00oW9r/ear0ozII3HvehPKmf8tiVqm9ywE3dmb7MEMH2R04G/wJ1oY ADFPXd+fM1/fniR6w+YM7Lj14RKKxMgi3Z0TLZpMA79HoUSWF6JRt1D9J3ejBqiT9dKq sxmA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id z3-v6si26822954pgh.557.2018.09.21.08.10.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:10:26 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:10:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71857024" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:09:54 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 26/27] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Date: Fri, 21 Sep 2018 08:03:50 -0700 Message-Id: <20180921150351.20898-27-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP arch_prctl(ARCH_CET_STATUS, unsigned long *addr) Return CET feature status. The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information: *addr = SHSTK/IBT status *(addr + 1) = SHSTK base address *(addr + 2) = SHSTK size arch_prctl(ARCH_CET_DISABLE, unsigned long features) Disable CET features specified in 'features'. Return -EPERM if CET is locked. arch_prctl(ARCH_CET_LOCK) Lock in CET feature. arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr) Allocate a new SHSTK. The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 27 +++++++++++ arch/x86/kernel/cet_prctl.c | 79 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 5 ++ 6 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kernel/cet_prctl.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index b7b33e1026bb..212bd68e31d3 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -12,19 +12,24 @@ struct task_struct; struct cet_status { unsigned long shstk_base; unsigned long shstk_size; + unsigned int locked:1; unsigned int shstk_enabled:1; }; #ifdef CONFIG_X86_INTEL_CET +int prctl_cet(int option, unsigned long arg2); int cet_setup_shstk(void); int cet_setup_thread_shstk(struct task_struct *p); +int cet_alloc_shstk(unsigned long *arg); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else +static inline int prctl_cet(int option, unsigned long arg2) { return 0; } static inline int cet_setup_shstk(void) { return 0; } static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } +static inline int cet_alloc_shstk(unsigned long *arg) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return 0; } diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 5a6aac9fa41f..3aec1088e01d 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -14,4 +14,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_CET_STATUS 0x3001 +#define ARCH_CET_DISABLE 0x3002 +#define ARCH_CET_LOCK 0x3003 +#define ARCH_CET_ALLOC_SHSTK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 36b14ef410c8..b9e6cdc6b4f7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,7 +139,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index ce0b3b7b1160..1c2689738604 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -110,6 +110,33 @@ static int create_rstor_token(bool ia32, unsigned long ssp, return 0; } +int cet_alloc_shstk(unsigned long *arg) +{ + unsigned long len = *arg; + unsigned long addr; + unsigned long token; + unsigned long ssp; + + addr = do_mmap_locked(0, len, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + if (addr >= TASK_SIZE_MAX) + return -ENOMEM; + + /* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + + if (!in_ia32_syscall()) + token |= 1; + ssp -= 8; + + if (write_user_shstk_64(ssp, token)) + return -EINVAL; + + *arg = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c new file mode 100644 index 000000000000..c4b7c19f5040 --- /dev/null +++ b/arch/x86/kernel/cet_prctl.c @@ -0,0 +1,79 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* See Documentation/x86/intel_cet.txt. */ + +static int handle_get_status(unsigned long arg2) +{ + unsigned int features = 0; + unsigned long shstk_base, shstk_size; + unsigned long buf[3]; + + if (current->thread.cet.shstk_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + + shstk_base = current->thread.cet.shstk_base; + shstk_size = current->thread.cet.shstk_size; + + buf[0] = (unsigned long)features; + buf[1] = shstk_base; + buf[2] = shstk_size; + return copy_to_user((unsigned long __user *)arg2, buf, + sizeof(buf)); +} + +static int handle_alloc_shstk(unsigned long arg2) +{ + int err = 0; + unsigned long shstk_size = 0; + + if (get_user(shstk_size, (unsigned long __user *)arg2)) + return -EFAULT; + + err = cet_alloc_shstk(&shstk_size); + if (err) + return err; + + if (put_user(shstk_size, (unsigned long __user *)arg2)) + return -EFAULT; + + return 0; +} + +int prctl_cet(int option, unsigned long arg2) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EINVAL; + + switch (option) { + case ARCH_CET_STATUS: + return handle_get_status(arg2); + + case ARCH_CET_DISABLE: + if (current->thread.cet.locked) + return -EPERM; + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + cet_disable_free_shstk(current); + + return 0; + + case ARCH_CET_LOCK: + current->thread.cet.locked = 1; + return 0; + + case ARCH_CET_ALLOC_SHSTK: + return handle_alloc_shstk(arg2); + + default: + return -EINVAL; + } +} diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 440f012ef925..251b8714f9a3 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -792,6 +792,11 @@ long do_arch_prctl_common(struct task_struct *task, int option, return get_cpuid_mode(); case ARCH_SET_CPUID: return set_cpuid_mode(task, cpuid_enabled); + case ARCH_CET_STATUS: + case ARCH_CET_DISABLE: + case ARCH_CET_LOCK: + case ARCH_CET_ALLOC_SHSTK: + return prctl_cet(option, cpuid_enabled); } return -EINVAL; From patchwork Fri Sep 21 15:03:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10610285 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8572215E8 for ; Fri, 21 Sep 2018 15:10:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6FFB72E44A for ; Fri, 21 Sep 2018 15:10:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61C032E4D8; Fri, 21 Sep 2018 15:10:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DDC82E4D8 for ; Fri, 21 Sep 2018 15:10:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68DEA8E0021; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5DD528E0020; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42D568E0021; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 02A0A8E001F for ; Fri, 21 Sep 2018 11:10:28 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id bg5-v6so6300282plb.20 for ; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=iY2ROvoUe5jFItNCy1Inb4JyspYsfHFmXNP33xUuHi91mU53cgBWc0NHmhjNo4iIg2 dyzV3xsBnLzkEyjehkHGWm3BVxKNq03MnIUCJWiliF0RD8jt1dE/nhWWtDAg5DVfC0lp 9OiGuLd0ZTp5SPNfpptnWCtPWqVXowknmS99Wrzz1VZ2OVxW4Id5ZBUyJk9oyYWqHHAE dLg475DqCANnGMKJRmiAegTlAom21xWsshEIzlEK9gnCCA+iUPA+ASmkWduT0rDGVu6z 7pEqtpm/qlpO+4sM1ttYcwS6iHbrZDkIAf3VWw7RhpanyyV3lwp58KrkERp+kR7vpUNc 7vmA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51ACw0JEHXSaqdKW/PIC6ij5knb07Z8zy5f9a6Tm9SlZy7VV30kS cv2wilzmFSIrqW2ugX62sqpZstvsXDVV+BOtivkh5vTQAvHN+U2SRUMbmNMMcH9LD4nIBzA/6Bv ZFV6N53/TQtQifeNN8VZPn9f6HA02iL1fCKZDYSXgXoCCpyRbwHpw8Le/bCOxBR2tDQ== X-Received: by 2002:a63:b705:: with SMTP id t5-v6mr39559007pgf.366.1537542627592; Fri, 21 Sep 2018 08:10:27 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZz61F1W2y7IAZqWOTeEKV8c3+3znJa1XCG3i078UDGK51ucRsY73k1a5Z7Zz2NiEQ56/v7 X-Received: by 2002:a63:b705:: with SMTP id t5-v6mr39558902pgf.366.1537542625968; Fri, 21 Sep 2018 08:10:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537542625; cv=none; d=google.com; s=arc-20160816; b=NqcssqZx2iYht9WerG0HMd5UVtBLNADxH9v/GZGf8Z1cLVyaBP5iYHNQEbW79SuqXr Xlsh2VfRlMNc+U5DfSQMdcR9uyXDlBYcGTKtXGaty48Tes0EkdjsMSx3GlvW7fBgsvmo 6Cwrmlpm+67Dock8pPlX4PbWge1KRZ5gOdPKZaoCLOKjFpjzuG1ncYKqtZ6Y2Op91Tjc BFxh4SC+1AmU+Rb0Dh7+1zg4lIougs5wiuxJiGRZGCklvhp11dSgmgbvP74BTwC35bYo 5AJMJijImVRX4QuEOYThPo7EwGNJ5pGx5aAtTbMhIFPlOHOLt2VRxhkNEKlO89b7gDJm 8YcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=F1oE/erK8M/WJJHqbtZ6wcU6z202nxjat6b9REe3aizgx5Zni7F9uN5owAjR6jFmW4 fFql1XJZO3ju9zKOpWEwhgSLg5bysE1aWD/S41gLu0bahGN8ThLqtIjTmdrQcpe0uiLC yWm3IQ76F1UJj7HA/S6qvv89PwSx6ufbAUcyy1su9avgcPaiUY4VXEmQsZtYS1KCtijZ ALFjf5qLswZ0VABNgJTIn9/hcBQO3hYNaHw8Tzw2BgNYg8InrFAy9M0JvTbKHm5wNwDX H+W69oQgVzTWHqbDgFGpGC3DTK4JxID/+whXotV05+txZz6RgxV6a9+obi8ghelFPj09 /eqg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id z3-v6si26822954pgh.557.2018.09.21.08.10.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 08:10:25 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Sep 2018 08:10:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,285,1534834800"; d="scan'208";a="71857055" Received: from 2b52.sc.intel.com ([143.183.136.51]) by fmsmga007.fm.intel.com with ESMTP; 21 Sep 2018 08:09:54 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v4 27/27] x86/cet/shstk: Add Shadow Stack instructions to opcode map Date: Fri, 21 Sep 2018 08:03:51 -0700 Message-Id: <20180921150351.20898-28-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180921150351.20898-1-yu-cheng.yu@intel.com> References: <20180921150351.20898-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the following shadow stack management instructions. INCSSP: Increment shadow stack pointer by the steps specified. RDSSP: Read SSP register into a GPR. SAVEPREVSSP: Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack. RSTORSSP: Restore from a "restore token" pointed by a GPR to SSP. WRSS: Write to kernel-mode shadow stack (kernel-mode instruction). WRUSS: Write to user-mode shadow stack (kernel-mode instruction). SETSSBSY: Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR. CLRSSBSY: Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable