From patchwork Thu Jun 6 20:06:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980365 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D58C1515 for ; Thu, 6 Jun 2019 20:15:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0BB90287C9 for ; Thu, 6 Jun 2019 20:15:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3E2F288DA; Thu, 6 Jun 2019 20:15:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 016BE287C9 for ; Thu, 6 Jun 2019 20:15:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A471D6B028A; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9CFA26B028B; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8742E6B028C; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 3D26F6B028A for ; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id o184so2614424pfg.1 for ; Thu, 06 Jun 2019 13:15:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=zJpnwswFcGz4As+pEnc6THBrdcqh4mpUN4Jp++oPJnI=; b=Z3bGdA9HtG+084l8ib+YP04dJx4lXLwoOyN0kvaFRGPReSercCqJL/SVZ23SnFChm2 LOEoXs14vB8DSgqPBfFqYZwMCOmwOr+1LNK3D448fTkeWMPzTYiCAmp5THj8uycGpGAN YS5FbfICfaDo5issu+Tf0ziWsaZVZHZk/N9QeVFT9S1Eb6tSwKk9pcEpA0VTq9TZrtLN /qSy+CH5Ac4tptiDv+c/jtTLu59Fe98whUgnJbat24UN6bxV3oMcUzCW8UfUYLgSrS65 JPCCM1r8m8mnow1shKHS1QgJ7zsOOOy4xRF/3mnx6la61Ah9W1ypSPUfGoYuATrcc6Wo rwVg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXKFCPrwI0IFt9TiVau/ieeENW2MWy6J3MrfCChPALrkL2XebR9 JBxvAnCBlXY3KOB5OeT81A9NEdds5EAVH2qklH5B+jVDFY3I5MaelapGCJLhOQMKfH0hfBjn4Zz xMm4fFmjySpP/sY2IFirKCaZN0EgMcLAt0E4JRRNi+B+4CTmRJVpGVd7X/MbDP4Oc9g== X-Received: by 2002:a63:4e10:: with SMTP id c16mr312629pgb.214.1559852111832; Thu, 06 Jun 2019 13:15:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqwGzVdy8+NcNJVX7rLRSP2EAbVldaXoMoHTgtGL1yPEP2IGyTIxKMgLBe2UTgSsy63b0s6/ X-Received: by 2002:a63:4e10:: with SMTP id c16mr312525pgb.214.1559852110257; Thu, 06 Jun 2019 13:15:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852110; cv=none; d=google.com; s=arc-20160816; b=agCsIlEKq/vB/zCz+Q+jdcUmZNyy10tTdukEauwQ10lottYAGiipihpZjqscTgWZXo lhJiyIfwRbYHKdKeCufcJjjRnA4s+jdNzgglwIbCBBzaevy1avsNfBkrJVfd1yqYw+vn +M6m5Fdx132x4pAv0lpRAY7ZoeikEETPhXuUkgJLPkKqX2W8gkJsnAx2rfAmpK25Ho4e vIlo/ph0xXK2QJUazbW0GVJur8XEqaWCN2sRHuYlM+fyWcdOWPaUfeX4YDJD/9ChqxyK vMfor8OuNRrLMg9vsUdi36owmP+jfgsLcZRDevowpVs50/gXqT50CjSCJluBUS5gCuxH i12A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=zJpnwswFcGz4As+pEnc6THBrdcqh4mpUN4Jp++oPJnI=; b=QN+j9yyDoDNGAOVVsR6tuh6o+nLe0rogLi3WZS8pusEGDt71cPymk1uqomNgBCEu3i GMszieb8jt13hjV515//mD1JM28iLtPxksdg+vF4zkCIu0LCMXfSDy1KHBhQqxoVw7ZS 4MesVpXaMCpQNXlNUHPhkwlNVkH1Wc5tniez/hJE8OZf56KRBPFP0rAvTEgdodfkcYpo YB/eYoVNgX4O5+eb8cvN/YlupcMfA2EPQ55YeisgV+pXmx6pERIv8tjLZxJfpOuBTfuf w69MrBoMwYMZzyGHpHM68psVM81+ukt//CUqmgUu40DLRO8KAqfg+5rGzy2jej9yBH5O AEoA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:10 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:09 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:08 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 01/27] Documentation/x86: Add CET description Date: Thu, 6 Jun 2019 13:06:20 -0700 Message-Id: <20190606200646.3951-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Explain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters. Signed-off-by: Yu-cheng Yu --- .../admin-guide/kernel-parameters.txt | 6 + Documentation/x86/index.rst | 1 + Documentation/x86/intel_cet.rst | 268 ++++++++++++++++++ 3 files changed, 275 insertions(+) create mode 100644 Documentation/x86/intel_cet.rst diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 138f6664b2e2..b9b054f6c732 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2906,6 +2906,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86,PPC] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst index ae36fc5fc649..47822ac02fe0 100644 --- a/Documentation/x86/index.rst +++ b/Documentation/x86/index.rst @@ -21,6 +21,7 @@ x86-specific Documentation pat protection-keys intel_mpx + intel_cet amd-memory-encryption pti mds diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst new file mode 100644 index 000000000000..dac83bbf8a24 --- /dev/null +++ b/Documentation/x86/intel_cet.rst @@ -0,0 +1,268 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================= +Control-flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control-flow Enforcement Technology (CET) provides protection against +return/jump-oriented programming (ROP) attacks. It can be setup to +protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented in 64-bit mode; 32-bit +applications are supported in compatibility mode. + +CET introduces shadow stack (SHSTK) and indirect branch tracking +(IBT). SHSTK is a secondary stack allocated from memory and cannot +be directly modified by applications. When executing a CALL, the +processor pushes a copy of the return address to SHSTK. Upon +function return, the processor pops the SHSTK copy and compares it +to the one from the program stack. If the two copies differ, the +processor raises a control-protection exception. IBT verifies all +indirect CALL/JMP targets are intended as marked by the compiler +with 'ENDBR' opcodes (see CET instructions below). + +There are two kernel configuration options: + + INTEL_X86_SHADOW_STACK_USER, and + INTEL_X86_BRANCH_TRACKING_USER. + +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later +are required. To build a CET-enabled application, GLIBC v2.28 or +later is also required. + +There are two command-line options for disabling CET features: + + no_cet_shstk - disables SHSTK, and + no_cet_ibt - disables IBT. + +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT. + +[2] CET assembly instructions +============================= + +RDSSP %r + Read the SHSTK pointer into %r. + +INCSSP %r + Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated + in the operand register. The GLIBC longjmp uses INCSSP to unwind + the SHSTK until that matches the program stack. When it is + necessary to unwind beyond 255 steps, longjmp divides and repeats + the process. + +RSTORSSP (%r) + Switch to the SHSTK indicated in the 'restore token' pointed by + the operand register and replace the 'restore token' with a new + token to be saved (with SAVEPREVSSP) for the outgoing SHSTK. + +:: + + Before RSTORSSP + + Incoming SHSTK Current/Outgoing SHSTK + + |----------------------| |----------------------| + addr=x | | ssp-> | | + |----------------------| |----------------------| + (%r)-> | rstor_token=(x|Lg) | addr=y-8 | | + |----------------------| |----------------------| + + After RSTORSSP + + |----------------------| |----------------------| + | | | | + |----------------------| |----------------------| + ssp-> | rstor_token=(y|Bz|Lg)| addr=y-8 | | + |----------------------| |----------------------| + + note: + 1. Only valid addresses and restore tokens can be on the + user-mode SHSTK. + 2. A token is always of type u64 and must align to u64. + 3. The incoming SHSTK pointer in a rstor_token must point to + immediately above the token. + 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK. + 5. 'Bz' is bit[1] of a rstor_token indicating the token is to + be used only for the next SAVEPREVSSP and invalid for the + RSTORSSP. + +SAVEPREVSSP + Store the SHSTK 'restore token' pointed by + (current_SHSTK_pointer + 8). + +:: + + After SAVEPREVSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) | + |----------------------| |----------------------| + +WRUSS %r0, (%r1) + Write the value in %r0 to the SHSTK address pointed by (%r1). + This is a kernel-mode only instruction. + +ENDBR + The compiler inserts an ENDBR at all valid branch targets. Any + CALL/JMP to a target without an ENDBR triggers a control + protection fault. + +[3] Application Enabling +======================== + +An application's CET capability is marked in its ELF header and can +be verified from the following command output, in the +NT_GNU_PROPERTY_TYPE_0 field: + + readelf -n + +If an application supports CET and is statically linked, it will run +with CET protection. If the application needs any shared libraries, +the loader checks all dependencies and enables CET only when all +requirements are met. + +[4] Legacy Libraries +==================== + +GLIBC provides a few tunables for backward compatibility. + +GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT + Turn off SHSTK/IBT for the current shell. + +GLIBC_TUNABLES=glibc.tune.x86_shstk= + This controls how dlopen() handles SHSTK legacy libraries: + on: continue with SHSTK enabled; + permissive: continue with SHSTK off. + +[5] CET system calls +==================== + +The following arch_prctl() system calls are added for CET: + +arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) + Return CET feature status. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = SHSTK/IBT status + *(addr + 1) = SHSTK base address + *(addr + 2) = SHSTK size + +arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) + Disable SHSTK and/or IBT specified in 'features'. Return -EPERM + if CET is locked. + +arch_prctl(ARCH_X86_CET_LOCK) + Lock in CET feature. + +arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) + Allocate a new SHSTK and put a restore token at top. + + The parameter 'addr' is a pointer to a user buffer and indicates + the desired SHSTK size to allocate. On returning to the caller, + the kernel fills *addr with the base address of the new SHSTK. + +arch_prctl(ARCH_X86_CET_SET_LEGACY_BITMAP, unsigned long *addr) + Setup an IBT legacy code bitmap. + + The parameter 'addr' is a pointer to a user buffer that has the + following information: + + *addr = IBT bitmap base address + *(addr + 1) = IBT bitmap size + +Note: + There is no CET enabling arch_prctl function. By design, CET is + enabled automatically if the binary and the system can support it. + + The parameters passed are always unsigned 64-bit. When an ia32 + application passing pointers, it should only use the lower 32 bits. + +[6] The implementation of the SHSTK +=================================== + +SHSTK size +---------- + +A task's SHSTK is allocated from memory to a fixed size of +RLIMIT_STACK. A compat-mode thread's SHSTK size is 1/4 of +RLIMIT_STACK. The smaller 32-bit thread SHSTK allows more threads to +share a 32-bit address space. + +Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, a large SHSTK will cover the +condition that both the program stack and the sigaltstack run out. + +The kernel creates a restore token at the SHSTK restoring address and +verifies that token when restoring from the signal handler. + +Fork +---- + +The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be +read-only and dirty. When a SHSTK PTE is not present, RO, and dirty, +a SHSTK access triggers a page fault with an additional SHSTK bit set +in the page fault error code. + +When a task forks a child, its SHSTK PTEs are copied and both the +parent's and the child's SHSTK PTEs are cleared of the dirty bit. +Upon the next SHSTK access, the resulting SHSTK page fault is handled +by page copy/re-use. + +When a pthread child is created, the kernel allocates a new SHSTK for +the new thread. + +Setjmp/Longjmp +-------------- + +Longjmp unwinds SHSTK until it matches the program stack. + +Ucontext +-------- + +In GLIBC, getcontext/setcontext is implemented in similar way as +setjmp/longjmp. + +When makecontext creates a new ucontext, a new SHSTK is allocated for +that context with ARCH_X86_CET_ALLOC_SHSTK the syscall. The kernel +creates a restore token at the top of the new SHSTK and the user-mode +code switches to the new SHSTK with the RSTORSSP instruction. + +[7] The management of read-only & dirty PTEs for SHSTK +====================================================== + +A RO and dirty PTE exists in the following cases: + +(a) A page is modified and then shared with a fork()'ed child; +(b) A R/O page that has been COW'ed; +(c) A SHSTK page. + +The processor only checks the dirty bit for (c). To prevent the use +of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as +DIRTY_SW for (a) and (b) above. This results to the following PTE +settings: + +Modified PTE: (R/W + DIRTY_HW) +Modified and shared PTE: (R/O + DIRTY_SW) +R/O PTE, COW'ed: (R/O + DIRTY_SW) +SHSTK PTE: (R/O + DIRTY_HW) +SHSTK PTE, COW'ed: (R/O + DIRTY_HW) +SHSTK PTE, shared: (R/O + DIRTY_SW) + +Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs. + +[8] The implementation of IBT +============================= + +The kernel provides IBT support in mmap() of the legacy code bit map. +However, the management of the bitmap is done in the GLIBC or the +application. From patchwork Thu Jun 6 20:06:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980367 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5EA551515 for ; Thu, 6 Jun 2019 20:15:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CA5A287C9 for ; Thu, 6 Jun 2019 20:15:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4069B288DA; Thu, 6 Jun 2019 20:15:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BADAA287C9 for ; Thu, 6 Jun 2019 20:15:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 121846B028B; Thu, 6 Jun 2019 16:15:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 00EE56B028D; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D781D6B028C; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 9DCF96B028D for ; Thu, 6 Jun 2019 16:15:12 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id e69so2309820pgc.7 for ; Thu, 06 Jun 2019 13:15:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7Czs0GMtnAfWbMOpCmbFxFl+kPT2QpCRlQ81FhMsGEA=; b=Jk0CvmUvUsyki87BSoZ9QKnYqjhP+wbmqPvud2kfBRwJ21dsxTLzyfS7m173x6Y7HU u1yRcd0SxF3OWqEdBUWfL1zLdjb9WbD30sG/BRxL+K4IBchltaw6qiBArK9ncrjLbiy7 4wfOP2sihCS/qnTdKxRy1+LTfHCjQG6LetBOOfOPgg80+0+9dkJZ27oP3+EYVEfXb4nd eyT5WaAAtZxVyo2+0qnrL5MZdKaOsQFyQLeRjH9rr6JmXliYcE7hFWPSNfsUfMzZieNA 1qLylxltk9hIFZVx+akDjG523rLifgUBHltvlIM+e2IFFsVCvEs9Msok/mIWzgqXGPLj zcTA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWDGOUulgrqQXhn6eGwy1JSYNpwfgtAqYw1eilyXf6eu64ZgaRC zx0oT52Da906ZTM0Ca3Hcs8H5odWAM+3dNcFJ16o1YV8uuo4oEcCK6xakRDOJguoo0N7rhQDp/c 7enfykTziqgucLIajqcyLgzgzAlaEsxPQDD7SuwS9Ap+4v0uhH28lHHiJCxJPj3NTCg== X-Received: by 2002:a63:470e:: with SMTP id u14mr293078pga.135.1559852112230; Thu, 06 Jun 2019 13:15:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqxcVfqNEWfs16llEt+Qkpid/2+bzUeI1Xs4iP5oRpR0uNjo64Me3bTcfGGs807wnnWpulFq X-Received: by 2002:a63:470e:: with SMTP id u14mr293025pga.135.1559852111426; Thu, 06 Jun 2019 13:15:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852111; cv=none; d=google.com; s=arc-20160816; b=Hhj6cSkylh7bJFU/FAZlUK10VOrJKoEH2Rv+IT731ByH2yC/T+KrH9dfP2ECNzX6Af z/gMq3g5lzOGhvmxVa8BqVi5+tbWncbw9p/Gct4KMmJf+GFe+v69KDhkosuP65NjDQqq U662JjbSp16FzC0ugw/GGo05NV+ghwVRG//CigKC7uK4cLNpYTEYPbYkcA4hsJx5rvkC KWxC/Q9Jiuyhb39dqb+WncsWyzkCb266RHd5ga2WJTUuhin34qyXCqH9W14/ZKEfytrk nFpuSxBQ9lJ0cjUHBsEQOQYk6DBbmMHKLVXcpeDTMivEQjIJRbGcBZaZs6sfHTZUMDWH lN8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7Czs0GMtnAfWbMOpCmbFxFl+kPT2QpCRlQ81FhMsGEA=; b=XVsCzIOQB4XrSDxDO6zO6HLYlVYCMlbviCkVV9RwlHHKByx+yAW6l7FRMU2Wni1sSN IOgvNFeECk1OyJbWoo1TXz4h64mUhkZo0BjfrAqWZjAsNV7UHrNmos9g4SW00M2QiRgd 8QaK5meq9o+a71bvSBM/FGonF6gdpQWpsZnCtB2JWW4ghVqqan977Z7WqumOArs7FNw2 8aBbb95tmWT4+uBJCeuIy2sUVAt4z9IOp5o9CmH5tg+S4rnfhIXHLSxWgFW2e+RYZhff GhS4lC9isLH9abRpBgeT0M2ap+Luy8OwIw6wlqUdLmiMOTBCk2rBsp+ltW8lvKMnAm8z R/Wg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:11 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:11 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:09 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 02/27] x86/cpufeatures: Add CET CPU feature flags for Control-flow Enforcement Technology (CET) Date: Thu, 6 Jun 2019 13:06:21 -0700 Message-Id: <20190606200646.3951-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add CPU feature flags for Control-flow Enforcement Technology (CET). CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking Signed-off-by: Yu-cheng Yu Reviewed-by: Borislav Petkov --- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 75f27ee2c263..21b2d9497c0f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -323,6 +323,7 @@ #define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ @@ -347,6 +348,7 @@ #define X86_FEATURE_MD_CLEAR (18*32+10) /* VERW clears CPU buffers */ #define X86_FEATURE_TSX_FORCE_ABORT (18*32+13) /* "" TSX_FORCE_ABORT */ #define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */ +#define X86_FEATURE_IBT (18*32+20) /* Indirect Branch Tracking */ #define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */ #define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */ #define X86_FEATURE_FLUSH_L1D (18*32+28) /* Flush L1D cache */ diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index 2c0bd38a44ab..68ef07175062 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -59,6 +59,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_AVX512_4VNNIW, X86_FEATURE_AVX512F }, { X86_FEATURE_AVX512_4FMAPS, X86_FEATURE_AVX512F }, { X86_FEATURE_AVX512_VPOPCNTDQ, X86_FEATURE_AVX512F }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, + { X86_FEATURE_IBT, X86_FEATURE_XSAVES }, {} }; From patchwork Thu Jun 6 20:06:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 824006C5 for ; Thu, 6 Jun 2019 20:15:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F926287C9 for ; Thu, 6 Jun 2019 20:15:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 632DA288DA; Thu, 6 Jun 2019 20:15:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EA57287C9 for ; Thu, 6 Jun 2019 20:15:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7E476B028D; Thu, 6 Jun 2019 16:15:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B326B6B028E; Thu, 6 Jun 2019 16:15:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 936186B028F; Thu, 6 Jun 2019 16:15:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 4EF246B028D for ; Thu, 6 Jun 2019 16:15:15 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id d7so2305733pgc.8 for ; Thu, 06 Jun 2019 13:15:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7V3NcjUGogy0AzDV0wQpP5vgz72/I1YSjIQMdGeAqKQ=; b=uTSCE9wQOsZwrXpDVNiXXY8C8qjIwwHxJL7TfPpw05ZuZNbFtISvaT2HvMOUGrxu5U imDEdUCPr0fDB2BVlmlGW3yhX2WaP3B7xMIzy35t+G9AQaqYxu0WJJ8tuByuD0DcX62w pKT7aGPSq0w7m+Z2mLw8+GjXrYSlfmeCFVEDqjB2n3yLI+aXEDmla3/ZXeqqXT/IE5HN certDz1PEAdf36edT9N7z2rcwTFO8APWI/DL/Qam2lA58KEDx/l6cOcMn+wJSlW+OjI2 lR9iVt+y2s8IZkKavf9BEkC7pAnNhCdSdXx/TWUxFx4ecF49AkrLst+nR1jrk7ZG6YWw hVVA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUd7T8JshO1oA7n2tAwJFwsqfrJ+yqT3tmE4Nx1jN1gcFv3jLbQ upJUk59pL7zTbYo5kg719h9yuc0NgVoBfs7qgRXBPnlzG00i44J2Bg2oF1tRFg+kWC35NYXGOVJ 0vJsbhfKZ3ndAmtaDv9K+wCwtbizgfuzQTGZflEuDrteYzoKrYYNqC+7SacJcxGSB4Q== X-Received: by 2002:a65:5241:: with SMTP id q1mr290286pgp.298.1559852114809; Thu, 06 Jun 2019 13:15:14 -0700 (PDT) X-Google-Smtp-Source: APXvYqwJP3l/r3Inb951ghZ5T9wSqFZPRsM5esShz4vxVWMxc45g4hznPLChnh3gQlOyK7Uo4/hG X-Received: by 2002:a65:5241:: with SMTP id q1mr290143pgp.298.1559852112853; Thu, 06 Jun 2019 13:15:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852112; cv=none; d=google.com; s=arc-20160816; b=poWbDreSztod9XE1WtKUpNt73DNmNLXjkdQe66Ag6SvpSu0pqH6H1Yrq1IKc73wra/ uo1RK+wzOTfk3riE6XM47txf0Z1M7vgWkDbdc1np1IfWOtP7dFLrYp/qKVBZl2K2cidv EQeZxXimrce75gX3Rg4tZbuZZkFiuAprjjxqimiSCIAqlzXHYsKLStVn47fj5ljcovz1 YYhxQnxumRkHXFJSbcznBQoSltVMLBJUjL5rr3Vr85D/SOzJJ0UHwUZkJly9IPZfZFI+ YTbPvwwD9J49gXY2P3/34vGmWutUa7P7izCfPzxXiF6Nci9IGdqKRcnvFwavyKYm6Xsi 9nFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7V3NcjUGogy0AzDV0wQpP5vgz72/I1YSjIQMdGeAqKQ=; b=pazr32BhLd6WPAU7UoTsAxQHQ+lp8GenME2z61uq58kgMjyg1C/Kk7ZIEUjh/HWLDO aUcIFWSt8ECydavyXmIi3CUPC+axBGZDnavPRQ3W2agJxPWHRs8XDmjqqO6M5ZSsWFBZ OIKMQoNjtBnuawm46nk2kwkthodifzRSWJ8bUjbCeWhVAGzYjuCqTHMJiK3QGIsNdirq ld0fRvltmyKDjvlR6rj87X2wLf6By+EgQLJJyyAq4QwlJ3aN4sYFA3C6kVMRzPuJjfr+ pteLJxZ6prlZ6PCuK9+pdtoYbdL80y8id0UEVjtgk36AZfbjcQlBIa10LWixD/E2HdYG XERQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:12 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:12 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:11 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 03/27] x86/fpu/xstate: Change names to separate XSAVES system and user states Date: Thu, 6 Jun 2019 13:06:22 -0700 Message-Id: <20190606200646.3951-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control-flow Enforcement (CET) MSR contents are XSAVES system states. To support CET, introduce XSAVES system states first. XSAVES is a "supervisor" instruction and, comparing to XSAVE, saves additional "supervisor" states that can be modified only from CPL 0. However, these states are per-task and not kernel's own. Rename "supervisor" states to "system" states to clearly separate them from "user" states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 4 +- arch/x86/include/asm/fpu/xstate.h | 20 +++---- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 10 ++-- arch/x86/kernel/fpu/xstate.c | 86 ++++++++++++++--------------- 5 files changed, 60 insertions(+), 62 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 9e27fa05a7ae..bcaa4fc54eb5 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -92,7 +92,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -225,7 +225,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) /* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact - * format and supervisor states in addition to modified optimization in + * format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 7e42b285c856..a1e33ebfb13f 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -25,15 +25,15 @@ #define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) /* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " @@ -41,7 +41,7 @@ #define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index ef0030e3fe6b..3271fd8b0322 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -230,7 +230,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ u64 __init fpu__get_supported_xfeatures_mask(void) { - return XCNTXT_MASK; + return SUPPORTED_XFEATURES_MASK; } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 5a8d118bc423..2aecbeaaee25 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -256,11 +256,11 @@ static int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_only) { if (use_xsave()) { if (fx_only) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_fxregs(buf); } else { - u64 init_bv = xfeatures_mask & ~xbv; + u64 init_bv = xfeatures_mask_user & ~xbv; if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_xregs(buf, xbv); @@ -359,7 +359,7 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) if (use_xsave() && !fx_only) { - u64 init_bv = xfeatures_mask & ~xfeatures; + u64 init_bv = xfeatures_mask_user & ~xfeatures; if (using_compacted_format()) { ret = copy_user_to_xstate(&fpu->state.xsave, buf_fx); @@ -390,7 +390,7 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) fpregs_lock(); if (use_xsave()) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); } @@ -464,7 +464,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 3c36dd1784db..d503e1fa15e8 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -52,13 +52,16 @@ static short xsave_cpuid_features[] __initdata = { }; /* - * Mask of xstate features supported by the CPU and the kernel: + * XSAVES system states can only be modified from CPL 0 and saved by + * XSAVES. The rest are user states. The following is a mask of + * supported user state features derived from boot_cpu_has() and + * SUPPORTED_XFEATURES_MASK. */ -u64 xfeatures_mask __read_mostly; +u64 xfeatures_mask_user __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -83,7 +86,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -114,15 +117,12 @@ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) } EXPORT_SYMBOL_GPL(cpu_has_xfeatures); -static int xfeature_is_supervisor(int xfeature_nr) +static int xfeature_is_system(int xfeature_nr) { /* - * We currently do not support supervisor states, but if - * we did, we could find out like this. - * * SDM says: If state component 'i' is a user state component, - * ECX[0] return 0; if state component i is a supervisor - * state component, ECX[0] returns 1. + * ECX[0] is 0; if state component i is a system state component, + * ECX[0] is 1. */ u32 eax, ebx, ecx, edx; @@ -132,7 +132,7 @@ static int xfeature_is_supervisor(int xfeature_nr) static int xfeature_is_user(int xfeature_nr) { - return !xfeature_is_supervisor(xfeature_nr); + return !xfeature_is_system(xfeature_nr); } /* @@ -165,7 +165,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask) == xfeatures_mask) + if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) return; /* @@ -192,7 +192,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * in a special way already: */ feature_bit = 0x2; - xfeatures = (xfeatures_mask & ~xfeatures) >> 2; + xfeatures = (xfeatures_mask_user & ~xfeatures) >> 2; /* * Update all the remaining memory layouts according to their @@ -220,20 +220,18 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) return; /* - * Make it clear that XSAVES supervisor states are not yet - * implemented should anyone expect it to work by changing - * bits in XFEATURE_MASK_* macros and XCR0. + * XCR_XFEATURE_ENABLED_MASK sets the features that are managed + * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be + * set here. */ - WARN_ONCE((xfeatures_mask & XFEATURE_MASK_SUPERVISOR), - "x86/fpu: XSAVES supervisor states are not yet implemented.\n"); - xfeatures_mask &= ~XFEATURE_MASK_SUPERVISOR; + xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -243,7 +241,7 @@ void fpu__init_cpu_xstate(void) */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask & (1UL << xfeature)); + return !!(xfeatures_mask_user & BIT_ULL(xfeature)); } /* @@ -273,7 +271,7 @@ static void __init setup_xstate_features(void) cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx); /* - * If an xfeature is supervisor state, the offset + * If an xfeature is a system state, the offset * in EBX is invalid. We leave it to -1. */ if (xfeature_is_user(i)) @@ -349,7 +347,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; int i; /* @@ -422,7 +420,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = (u64)1 << 63 | xfeatures_mask; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; /* * Init all the features state with header.xfeatures being 0x0 @@ -441,8 +439,8 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports supervisor states and it uses compacted - * format. Checking a supervisor state's uncompacted offset is + * Only XSAVES supports system states and it uses compacted + * format. Checking a system state's uncompacted offset is * an error. */ if (XFEATURE_MASK_SUPERVISOR & BIT_ULL(xfeature_nr)) { @@ -466,7 +464,7 @@ static int xfeature_size(int xfeature_nr) /* * 'XSAVES' implies two different things: - * 1. saving of supervisor/system state + * 1. saving of system state * 2. using the compacted format * * Use this function when dealing with the compacted format so @@ -481,8 +479,8 @@ int using_compacted_format(void) /* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ int validate_xstate_header(const struct xstate_header *hdr) { - /* No unknown or supervisor features may be set */ - if (hdr->xfeatures & (~xfeatures_mask | XFEATURE_MASK_SUPERVISOR)) + /* No unknown or system features may be set */ + if (hdr->xfeatures & ~xfeatures_mask_user) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -589,11 +587,11 @@ static void do_extra_xstate_size_checks(void) check_xstate_against_struct(i); /* - * Supervisor state components can be managed only by + * System state components can be managed only by * XSAVES, which is compacted-format only. */ if (!using_compacted_format()) - XSTATE_WARN_ON(xfeature_is_supervisor(i)); + XSTATE_WARN_ON(xfeature_is_system(i)); /* Align from the end of the previous feature */ if (xfeature_is_aligned(i)) @@ -617,7 +615,7 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask. + * Get total size of enabled xstates in XCR0/xfeatures_mask_user. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer @@ -707,7 +705,7 @@ static int __init init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { - xfeatures_mask = 0; + xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); } @@ -743,15 +741,15 @@ void __init fpu__init_system_xstate(void) } cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask = eax + ((u64)edx << 32); + xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue * booting without it. This is too early to BUG(). */ - pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask); + pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask_user); goto out_disable; } @@ -760,10 +758,10 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask &= ~BIT(i); + xfeatures_mask_user &= ~BIT_ULL(i); } - xfeatures_mask &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -773,9 +771,9 @@ void __init fpu__init_system_xstate(void) /* * Update info used for ptrace frames; use standard-format size and no - * supervisor xstates: + * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -783,7 +781,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask, + xfeatures_mask_user, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -802,7 +800,7 @@ void fpu__resume_cpu(void) * Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has(X86_FEATURE_XSAVE)) - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -850,7 +848,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) * have not enabled. Remember that pcntxt_mask is * what we write to the XCR0 register. */ - WARN_ONCE(!(xfeatures_mask & BIT_ULL(xfeature_nr)), + WARN_ONCE(!(xfeatures_mask_user & BIT_ULL(xfeature_nr)), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to From patchwork Thu Jun 6 20:06:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980371 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75D5613AD for ; Thu, 6 Jun 2019 20:15:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 632E7287C9 for ; Thu, 6 Jun 2019 20:15:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 566D4288DA; Thu, 6 Jun 2019 20:15:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BCAC287C9 for ; Thu, 6 Jun 2019 20:15:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D79FD6B028E; Thu, 6 Jun 2019 16:15:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D0A5A6B028F; Thu, 6 Jun 2019 16:15:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B81E76B0290; Thu, 6 Jun 2019 16:15:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 765EC6B028E for ; Thu, 6 Jun 2019 16:15:16 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id o12so2142947pll.17 for ; Thu, 06 Jun 2019 13:15:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ZVVWngZ2vi1M+l8eju2NuXAqNb36mQe0Pek1Z/FGLms=; b=HNuRLGYTdWWpAJugHRmA2DH6KFAI4sL9AnlQQCL2xn6KOKL0XGplB4r1zGYxyuFkbu 3eE68MIa8nuL/Q/SXwuUhQl+9v+SvMbb1lXWYFV5XHqCjf0enB65sBRUulb6oWgN2SgO hdY9rIWcd6PEXI6nWFv31vHjES+5VI0/seBBBSZYqaFbNY9gR4lm7gc+frYsHXiGHeUv uxE++I/L6b7y6pBmoivD4ar2OXp16pfCJvwasNG7UgWTIhmjkIUDeIKPVj5BjZkdbbU1 f+MnnO9fIoK7WFgX9g5wJ5fS+lYUSpE96p+Q1T3BAUl8mi5hS2a/O/oB33U6qCkG59AD q2Lw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVKop7dySzWzlc7cRb/S+JTnoyXjukDLGhG1tA1vV4p3xNMRhhE c7+s2y3FkrA+6X/RGmkTyIM4vzq5JI/8zySgCPwTXAtIE3Ms8lOh7fHrFPPCT/tPofef27iV2Ms 5fafy4v+ZQxrMj2741lV5cPMmOtvc+BK30J158qoLndlLmcp+jG/e2pXJaYvyq2h4LA== X-Received: by 2002:a63:788a:: with SMTP id t132mr339143pgc.52.1559852116058; Thu, 06 Jun 2019 13:15:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqygjfvo9xYFRxdlVKUe2lr5ztzGChOuS/zsOAjICzLlbCGYkiInQUQyik2P24jQUirxFaBe X-Received: by 2002:a63:788a:: with SMTP id t132mr339016pgc.52.1559852114253; Thu, 06 Jun 2019 13:15:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852114; cv=none; d=google.com; s=arc-20160816; b=Ry0qhOPfUa9cVxQ05eEaYDLKEqOn1CJYmEd918Cn5r7TRvKPl0mYOAHLQWCe78eiFl amj7pdju6A0CaQxrIU8OtOeMyRCPGTV8wKKiQKtiU+a1FppMSj1oH4jnlR6YsAc4Bf4L GQQX/Fe/RqmZfiSu5MZXYUiN6j97hiMrP8yrLGqtumf20cjhJ4CKzCcPhlN6qa4bK1Kr Ipn8fgYnLpqynZR1/cHpYuVkJyyv/dYbaNa8b+OMGhdv27W+FlmlAqj1i2mOTJgIHtTU nzhLiym4qd6Y9MlCcdbeVaInJO1QSnAhFss8ek269TEuw/hdkGOxgE0WhgNxoVIvU82M oRbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ZVVWngZ2vi1M+l8eju2NuXAqNb36mQe0Pek1Z/FGLms=; b=akGbEBXzbribHVf7NQzNl+uhqSQ9N1Y7YeXpOvALRdVOAZ1CZzXwKLZ6n6Ww2dpPQM l6oFd6Y4rldqXwJoNE9bkMI5/CsWadA9ZygdA9rKfEi4cdaB+xAlAiAbwtmglnVVIFdL AkE/0Ex5RewPWVvq3gg2gUW7Rv46aobQDcicKcSu3eMuOzeNujGcA0oLbAQ/h9veyW44 pWti/J7XI9nN9GwgMHEHI2eJB0FwK0bGSm0lplkkCy6vhzxi8gNM/B9xkFRWpio6ozyz ZHEmcnzhVLMcRl6E3NsGs7+qkviteBAuSDMkmfw6X0FHtq3EISehGJ/apMWJbuHAZpPD qitQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:14 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:13 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:12 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 04/27] x86/fpu/xstate: Introduce XSAVES system states Date: Thu, 6 Jun 2019 13:06:23 -0700 Message-Id: <20190606200646.3951-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control-flow Enforcement (CET) MSR contents are XSAVES system states. To support CET, introduce XSAVES system states first. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 21 ++++++- arch/x86/include/asm/fpu/xstate.h | 4 +- arch/x86/kernel/fpu/core.c | 26 +++++++-- arch/x86/kernel/fpu/init.c | 10 ---- arch/x86/kernel/fpu/signal.c | 4 +- arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++++++---------- arch/x86/kernel/process.c | 2 +- arch/x86/kernel/signal.c | 2 +- 8 files changed, 104 insertions(+), 55 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index bcaa4fc54eb5..148a3d8c8c35 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -31,7 +31,8 @@ extern void fpu__save(struct fpu *fpu); extern int fpu__restore_sig(void __user *buf, int ia32_frame); extern void fpu__drop(struct fpu *fpu); extern int fpu__copy(struct task_struct *dst, struct task_struct *src); -extern void fpu__clear(struct fpu *fpu); +extern void fpu__clear_user_states(struct fpu *fpu); +extern void fpu__clear_all(struct fpu *fpu); extern int fpu__exception_code(struct fpu *fpu, int trap_nr); extern int dump_fpu(struct pt_regs *ptregs, struct user_i387_struct *fpstate); @@ -44,7 +45,6 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system(struct cpuinfo_x86 *c); extern void fpu__init_check_bugs(void); extern void fpu__resume_cpu(void); -extern u64 fpu__get_supported_xfeatures_mask(void); /* * Debugging facility: @@ -92,7 +92,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -615,6 +615,21 @@ static inline void switch_fpu_finish(struct fpu *new_fpu) __write_pkru(pkru_val); } +/* + * Helpers for changing XSAVES system states. + */ +static inline void modify_fpu_regs_begin(void) +{ + fpregs_lock(); + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + __fpregs_load_activate(); +} + +static inline void modify_fpu_regs_end(void) +{ + fpregs_unlock(); +} + /* * MXCSR and XCR definitions: */ diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index a1e33ebfb13f..2ec19415c58e 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -21,9 +21,6 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) - /* All currently supported features */ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ @@ -42,6 +39,7 @@ #endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 466fca686fb9..7a9dcd69580a 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -316,12 +316,16 @@ void fpu__drop(struct fpu *fpu) * Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs(void) +static inline void copy_init_fpstate_to_fpregs(u64 features_mask) { fpregs_lock(); + /* + * Only XSAVES user states are copied. + * System states are preserved. + */ if (use_xsave()) - copy_kernel_to_xregs(&init_fpstate.xsave, -1); + copy_kernel_to_xregs(&init_fpstate.xsave, features_mask); else if (static_cpu_has(X86_FEATURE_FXSR)) copy_kernel_to_fxregs(&init_fpstate.fxsave); else @@ -340,7 +344,21 @@ static inline void copy_init_fpstate_to_fpregs(void) * Called by sys_execve(), by the signal handler code and by various * error paths. */ -void fpu__clear(struct fpu *fpu) +void fpu__clear_user_states(struct fpu *fpu) +{ + WARN_ON_FPU(fpu != ¤t->thread.fpu); /* Almost certainly an anomaly */ + + fpu__drop(fpu); + + /* + * Make sure fpstate is cleared and initialized. + */ + fpu__initialize(fpu); + if (static_cpu_has(X86_FEATURE_FPU)) + copy_init_fpstate_to_fpregs(xfeatures_mask_user); +} + +void fpu__clear_all(struct fpu *fpu) { WARN_ON_FPU(fpu != ¤t->thread.fpu); /* Almost certainly an anomaly */ @@ -351,7 +369,7 @@ void fpu__clear(struct fpu *fpu) */ fpu__initialize(fpu); if (static_cpu_has(X86_FEATURE_FPU)) - copy_init_fpstate_to_fpregs(); + copy_init_fpstate_to_fpregs(xfeatures_mask_all); } /* diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 3271fd8b0322..d9b2e8ec4e4b 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -223,16 +223,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* - * Find supported xfeatures based on cpu features and command-line input. - * This must be called after fpu__init_parse_early_param() is called and - * xfeatures_mask is enumerated. - */ -u64 __init fpu__get_supported_xfeatures_mask(void) -{ - return SUPPORTED_XFEATURES_MASK; -} - /* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch(void) { diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 2aecbeaaee25..e38b272793b1 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -287,7 +287,7 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) IS_ENABLED(CONFIG_IA32_EMULATION)); if (!buf) { - fpu__clear(fpu); + fpu__clear_user_states(fpu); return 0; } @@ -409,7 +409,7 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) err_out: if (ret) - fpu__clear(fpu); + fpu__clear_user_states(fpu); return ret; } diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index d503e1fa15e8..6b453455a4f0 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -59,9 +59,19 @@ static short xsave_cpuid_features[] __initdata = { */ u64 xfeatures_mask_user __read_mostly; +/* + * Supported XSAVES system states. + */ +static u64 xfeatures_mask_system __read_mostly; + +/* + * Combined XSAVES system and user states. + */ +u64 xfeatures_mask_all __read_mostly; + static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_all)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -86,7 +96,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -165,7 +175,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) + if ((xfeatures & xfeatures_mask_all) == xfeatures_mask_all) return; /* @@ -220,28 +230,27 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_all) return; /* * XCR_XFEATURE_ENABLED_MASK sets the features that are managed * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be * set here. */ - - xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; - cr4_set_bits(X86_CR4_OSXSAVE); xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * MSR_IA32_XSS controls which system (not user) states are + * to be managed by XSAVES. + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_system); } -/* - * Note that in the future we will likely need a pair of - * functions here: one for user xstates and the other for - * system xstates. For now, they are the same. - */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask_user & BIT_ULL(xfeature)); + return !!(xfeatures_mask_all & BIT_ULL(xfeature)); } /* @@ -347,7 +356,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_all)*8]; int i; /* @@ -420,7 +429,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_all; /* * Init all the features state with header.xfeatures being 0x0 @@ -443,7 +452,7 @@ static int xfeature_uncompacted_offset(int xfeature_nr) * format. Checking a system state's uncompacted offset is * an error. */ - if (XFEATURE_MASK_SUPERVISOR & BIT_ULL(xfeature_nr)) { + if (~xfeatures_mask_user & BIT_ULL(xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -615,15 +624,12 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask_user. + * Get total size of enabled xstates in XCR0 | IA32_XSS. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer * that we use 'XSAVES' on, we could potentially overflow the * buffer because 'XSAVES' saves system states too. - * - * Note that we do not currently set any bits on IA32_XSS so - * 'XCR0 | IA32_XSS == XCR0' for now. */ static unsigned int __init get_xsaves_size(void) { @@ -705,6 +711,7 @@ static int __init init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { + xfeatures_mask_all = 0; xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); @@ -740,10 +747,23 @@ void __init fpu__init_system_xstate(void) return; } + /* + * Find user states supported by the processor. + * Only these bits can be set in XCR0. + */ cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + /* + * Find system states supported by the processor. + * Only these bits can be set in IA32_XSS MSR. + */ + cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx); + xfeatures_mask_system = ecx + ((u64)edx << 32); + + xfeatures_mask_all = xfeatures_mask_user | xfeatures_mask_system; + + if ((xfeatures_mask_all & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -758,10 +778,12 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask_user &= ~BIT_ULL(i); + xfeatures_mask_all &= ~BIT_ULL(i); } - xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_all &= SUPPORTED_XFEATURES_MASK; + xfeatures_mask_user &= xfeatures_mask_all; + xfeatures_mask_system &= xfeatures_mask_all; /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -773,7 +795,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -781,7 +803,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask_user, + xfeatures_mask_all, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -801,6 +823,12 @@ void fpu__resume_cpu(void) */ if (boot_cpu_has(X86_FEATURE_XSAVE)) xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * Restore IA32_XSS + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_system); } /* @@ -846,9 +874,9 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr) /* * We should not ever be requesting features that we * have not enabled. Remember that pcntxt_mask is - * what we write to the XCR0 register. + * what we write to the XCR0 | IA32_XSS registers. */ - WARN_ONCE(!(xfeatures_mask_user & BIT_ULL(xfeature_nr)), + WARN_ONCE(!(xfeatures_mask_all & BIT_ULL(xfeature_nr)), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -996,7 +1024,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1080,7 +1108,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1173,7 +1201,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= xfeatures_mask_system; /* * Add back in the features that came in from userspace: @@ -1229,7 +1257,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= xfeatures_mask_system; /* * Add back in the features that came in from userspace: diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 75fea0d48c0e..d360bf4d696b 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -139,7 +139,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(tsk); memset(tsk->thread.tls_array, 0, sizeof(tsk->thread.tls_array)); - fpu__clear(&tsk->thread.fpu); + fpu__clear_all(&tsk->thread.fpu); } void disable_TSC(void) diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 364813cea647..3b0dcec597ce 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -763,7 +763,7 @@ handle_signal(struct ksignal *ksig, struct pt_regs *regs) /* * Ensure the signal handler starts with the new fpu state. */ - fpu__clear(fpu); + fpu__clear_user_states(fpu); } signal_setup_done(failed, ksig, stepping); } From patchwork Thu Jun 6 20:06:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980373 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F2291515 for ; Thu, 6 Jun 2019 20:15:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DE62287C9 for ; Thu, 6 Jun 2019 20:15:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81816288DA; Thu, 6 Jun 2019 20:15:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E2EFD287C9 for ; Thu, 6 Jun 2019 20:15:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FA4F6B028F; Thu, 6 Jun 2019 16:15:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9357A6B0292; Thu, 6 Jun 2019 16:15:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7598A6B0293; Thu, 6 Jun 2019 16:15:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 291D06B028F for ; Thu, 6 Jun 2019 16:15:17 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id z10so2291664pgf.15 for ; Thu, 06 Jun 2019 13:15:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=xNZB2zLiZ21eOdSBBflS3RlJecfHCk2sycG3I8os74o=; b=HOJ4jwvVW24JE6BAR7jlTC+g6hNy+eiYMpOi0Kt1U6WOEWqwOO8OZCFcdSf5zBL2fY 62WBZLpnfHV2HkRzHRVS5UoM9fVCO2jTjZAA64ZGDhVRBWpyyE1XI3h99emUht1dsSAy IqExd0gg4QDXGbNaShBYjKucKlPDVzA4KayoReRAY3auhRlGP4SdVrYX8fV5+pvcePZw sODRZJrDffEukjcn+lj1pXiow7FLBuQ4zFxCPgQ4UcqOMl3CEzBDboK8f4rjGGYzhcfl rwDZYa/z+ZENGsAiiz/zHp8jb/nPtfTxW893QFL7t52Y2huNZhAth4GYY7SPPfsmGWz0 VB8A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXmIWEFYR6csMWSuqxjGm7+kWmzUw268GLkBFXtHPMm+C6782wo y6jSj+0tWtfWe1yUPhL/bb0XCUOXBlDRg0cFL7fGR2RMJZgGLYzVdqoi+2L8BokINE8KMZGuIf6 hZoC8+YQsYELW3puvgq0dNa2/DduhFH2U/enXUM7IvFGtU6sbf323O0Ca6rEqVvOocQ== X-Received: by 2002:a17:90a:2561:: with SMTP id j88mr1637636pje.121.1559852116818; Thu, 06 Jun 2019 13:15:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqwvNs7rsDrIikOlkllqgy4G0quYo3NlNkdkB+r+ixYDlm8vYT3fGV2I5PLwnHzxCb6l5w0u X-Received: by 2002:a17:90a:2561:: with SMTP id j88mr1637529pje.121.1559852115423; Thu, 06 Jun 2019 13:15:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852115; cv=none; d=google.com; s=arc-20160816; b=HsVtDua7yqQhCBWS/TckusnnNpuHULlswjmd0tS9YnkQFq8rsjCKShVcpLFRwnXeCE HMtFWjscADuvV2m/dLKACA5hhYYLGytP/lbcTNbO6CTulm+xxuagnglZUfclAcBKTWpz Zm5L7Yz7g3cPrV42gxuCzNDG4STJ/pSOQy/yincaga7ajskUf6Sq1akSwhkR4Zx+nhr1 rDofmAn9F6CbYaHJaP3Hqho6UEpb6lrpYG8BQcmNUu0G5jWfY/114J61YbTDASzoQ+sY eT/z908/YRb1c5BYjV1oPd/XIhGcJoqUPQ1w/c/Y5yUQufjFldVl9xl+p6x7sTnaIpvI rR1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=xNZB2zLiZ21eOdSBBflS3RlJecfHCk2sycG3I8os74o=; b=xJVAhvtTZ/VikvIb5mmr33znv++UjnmG3JElCIBZxX0UyBw9A3zfyPRqJrwhaYm+tx 5/atSeGNfl0HUoZjoYRxIB2fVK1euPkRerHbNycHFbngiQUbHXD+yb9Ils3NWxkpriu/ A6oZcz/oz+hZlOcSbtgPmRCjvAzZnVboom81TKPgyojBjdqy7HW9/VHQKqCfQbXH3DZh Hj4HORkWMy25hDwn7RxLrNtUQYlRrAzdls0zOtMrXodJeranX6U6gz8aOYzONECIYne2 1k+8X/bH+vyWvxfP823WHrV2m0ki0R+/LvgXmKFKYO7TaxW004HHSPokrzvEF67jafY2 Gs2w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:15 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:14 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:13 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 05/27] x86/fpu/xstate: Add XSAVES system states for shadow stack Date: Thu, 6 Jun 2019 13:06:24 -0700 Message-Id: <20190606200646.3951-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Intel Control-flow Enforcement Technology (CET) introduces the following MSRs. MSR_IA32_U_CET (user-mode CET settings), MSR_IA32_PL3_SSP (user-mode shadow stack), MSR_IA32_PL0_SSP (kernel-mode shadow stack), MSR_IA32_PL1_SSP (Privilege Level 1 shadow stack), MSR_IA32_PL2_SSP (Privilege Level 2 shadow stack). Introduce them into XSAVES system states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index f098f6cab94b..d7ef4d9c7ad5 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM, XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, + XFEATURE_RESERVED, + XFEATURE_CET_USER, + XFEATURE_CET_KERNEL, XFEATURE_MAX, }; @@ -128,6 +131,8 @@ enum xfeature { #define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM) #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL) #define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE) #define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \ @@ -229,6 +234,23 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control-flow Enforcement user states + */ +struct cet_user_state { + u64 user_cet; /* user control-flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +}; + +/* + * State component 12 is Control-flow Enforcement kernel states + */ +struct cet_kernel_state { + u64 kernel_ssp; /* kernel shadow stack */ + u64 pl1_ssp; /* privilege level 1 shadow stack */ + u64 pl2_ssp; /* privilege level 2 shadow stack */ +}; + struct xstate_header { u64 xfeatures; u64 xcomp_bv; diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 2ec19415c58e..9ac8a81e851d 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -30,7 +30,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_CET_USER | \ + XFEATURE_MASK_CET_KERNEL) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..a8df907e8017 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -130,6 +130,8 @@ #define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT) #define X86_CR4_PKE_BIT 22 /* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) /* * x86-64 Task Priority Register, CR8 diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 6b453455a4f0..7f99878111d7 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -36,6 +36,9 @@ static const char *xfeature_names[] = "Processor Trace (unused)" , "Protection Keys User registers", "unknown xstate feature" , + "Control-flow User registers" , + "Control-flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = { @@ -49,6 +52,9 @@ static short xsave_cpuid_features[] __initdata = { X86_FEATURE_AVX512F, X86_FEATURE_INTEL_PT, X86_FEATURE_PKU, + 0, /* Unused */ + X86_FEATURE_SHSTK, /* XFEATURE_CET_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_CET_KERNEL */ }; /* @@ -320,6 +326,8 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_ZMM_Hi256); print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); + print_xstate_feature(XFEATURE_MASK_CET_USER); + print_xstate_feature(XFEATURE_MASK_CET_KERNEL); } /* @@ -566,6 +574,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_CET_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_CET_KERNEL, struct cet_kernel_state); /* * Make *SURE* to add any feature numbers in below if From patchwork Thu Jun 6 20:06:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FA571515 for ; Thu, 6 Jun 2019 20:15:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5FE6C287C9 for ; Thu, 6 Jun 2019 20:15:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 512EE288DA; Thu, 6 Jun 2019 20:15:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9054A287C9 for ; Thu, 6 Jun 2019 20:15:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31B3E6B0292; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2AA266B0293; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05E5B6B0294; Thu, 6 Jun 2019 16:15:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id B6ACF6B0292 for ; Thu, 6 Jun 2019 16:15:18 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id d125so2614589pfd.3 for ; Thu, 06 Jun 2019 13:15:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Agqu7L2UI0Tkjxyheb61lr04vYuCi6Sfe41rIBPXzL4=; b=pHvhlTrpHJmDTbV5rwKC413NPnr3V+OMnQDSnth7CcGknNC2Z678R879YjtngWLkV/ BcS8O8NBru5Qkx13niwNugQKDOM58B1nbvq7Eimn8owmhFc/NdJbCBwqDk945WKOUNyK HNWEP4HofzlRP+rt+QI57OnYx75n+XOgZucIWc5dGBkCsreMzr05egn8GOCiqmi4TSNv lHrAsXeeT+anzlg1cjRiqRxtF1CrDgZaZNUobrdNK3b3XDF3LtUtPLqxdRAv26C4KtvD Yc5gYRavk9yZ7da0Hc39HrV9gCzGesJFzlbH707QQnHZYdQEXPhw4/bI/7Owvk6LXBqD wOnQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXETrSscTdJFJUfv0tPu+AQTyAABwa0RIHbvgSngF0B03prs/80 4ore+i3BAIU6St7pA+d45+itY80MGh7WFRxQNG9B7XSGJx0lUo+TJBdHN5v7wOUOEDSdQwMJ70B e71lT7fzRV6dZzz09hb25B25RUp951yCsmVESERuW1XKKOt9fDomiq9tSZmtBQ+gMNw== X-Received: by 2002:a65:56c2:: with SMTP id w2mr299367pgs.49.1559852118208; Thu, 06 Jun 2019 13:15:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqz5PEfxW/FfsqdztvOtU1XbBxaGUN2sQ7bRZdnRgp73AjDaGCTpuZfpUBcFwTtMAfeuhA3L X-Received: by 2002:a65:56c2:: with SMTP id w2mr299276pgs.49.1559852116842; Thu, 06 Jun 2019 13:15:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852116; cv=none; d=google.com; s=arc-20160816; b=u08zvOlJwxLcfwTPs/W54byvtzsIq4Ov5F6kTksH9JHz0UcNJWUByblucbwEsS88if 7lho/mkXd99x2ourDYQ9JUAPGWmASLi4g/I3laJtfzxgW3ubeZ2AemK8D5wcOVP9eHa9 RjdFK3+bpWsiB8ySXJ2SsEj1dRqxcAUjuFS/4ehvqqzNAraVrO6mEmCFvgkBxicDYsjH x38RjGPWq+nouU2R6uNFFBogc7nXfaRowup1Cd8UiOBksx7KybltZ17TSZquJsA+DjWK p9qhXECP28HFylcrkO58EKVs4OZE9T5ONv9eCvxdlsZbT6HAutCkcZVfUrAhbToG2kMB MjBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Agqu7L2UI0Tkjxyheb61lr04vYuCi6Sfe41rIBPXzL4=; b=reaQrn2HxDGKDfJj2IObEsYII3VLHbxYsw26g/Vt4D4JEBExiYfZ4uZyL1nVARvd1C S674YFQuPbQjhBXNA5wWd3OsOGUp8dcvG0Ns8ArvbV5p2jVGO+EHtrDkJCA/zTzfppuP 0PW1Y7XARCn6GCku3Dvdy93pwGzHSCv9ewFfA1YR0kZsOG0tT32K6FqPZdDk4ZXhbFka XW/kwrYOLMqy57CrgRSPGkDgm/6aFzvqnLtYpt2wogaKNeCbUZ4ScX5TJysA+Ek8jeK2 JprTedp+TmXY2Ny2jVlsg/WkSd1EgqfMFz3FpZcHgaTrvyTQ4NFqzgpIrLcKbjOx9wWd YRgw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:16 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:16 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:15 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 06/27] x86/cet: Add control protection exception handler Date: Thu, 6 Jun 2019 13:06:25 -0700 Message-Id: <20190606200646.3951-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction. The control protection exception handler works in a similar way as the general protection fault handler. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/signal_compat.c | 2 +- arch/x86/kernel/traps.c | 57 ++++++++++++++++++++++++++++++ include/uapi/asm-generic/siginfo.h | 3 +- 6 files changed, 68 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 11aa3b2afa4d..595c2efbb893 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -993,7 +993,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 7d6f3f3fad78..5906a22796b6 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void); @@ -81,6 +82,7 @@ struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s); void __init trap_init(void); #endif dotraplinkage void do_general_protection(struct pt_regs *regs, long error_code); +dotraplinkage void do_control_protection(struct pt_regs *regs, long error_code); dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *regs, long error_code); dotraplinkage void do_coprocessor_error(struct pt_regs *regs, long error_code); @@ -151,6 +153,7 @@ enum { X86_TRAP_AC, /* 17, Alignment Check */ X86_TRAP_MC, /* 18, Machine Check */ X86_TRAP_XF, /* 19, SIMD Floating-Point Exception */ + X86_TRAP_CP = 21, /* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ }; diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 6d8917875f44..588848d00fff 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -103,6 +103,10 @@ static const __initconst struct idt_data def_idts[] = { #elif defined(CONFIG_X86_32) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32), #endif + +#ifdef CONFIG_X86_64 + INTG(X86_TRAP_CP, control_protection), +#endif }; /* diff --git a/arch/x86/kernel/signal_compat.c b/arch/x86/kernel/signal_compat.c index 9ccbf0576cd0..c572a3de1037 100644 --- a/arch/x86/kernel/signal_compat.c +++ b/arch/x86/kernel/signal_compat.c @@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void) */ BUILD_BUG_ON(NSIGILL != 11); BUILD_BUG_ON(NSIGFPE != 15); - BUILD_BUG_ON(NSIGSEGV != 7); + BUILD_BUG_ON(NSIGSEGV != 8); BUILD_BUG_ON(NSIGBUS != 5); BUILD_BUG_ON(NSIGTRAP != 5); BUILD_BUG_ON(NSIGCHLD != 6); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 8b6d03e55d2f..db143d447bba 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -570,6 +570,63 @@ do_general_protection(struct pt_regs *regs, long error_code) } NOKPROBE_SYMBOL(do_general_protection); +static const char *control_protection_err[] = { + "unknown", + "near-ret", + "far-ret/iret", + "endbranch", + "rstorssp", + "setssbsy", +}; + +/* + * When a control protection exception occurs, send a signal + * to the responsible application. Currently, control + * protection is only enabled for the user mode. This + * exception should not come from the kernel mode. + */ +dotraplinkage void +do_control_protection(struct pt_regs *regs, long error_code) +{ + struct task_struct *tsk; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); + if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code, X86_TRAP_CP, SIGSEGV) == NOTIFY_STOP) + return; + cond_local_irq_enable(regs); + + if (!user_mode(regs)) + die("kernel control protection fault", regs, error_code); + + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + printk_ratelimit()) { + unsigned int max_err; + + max_err = ARRAY_SIZE(control_protection_err) - 1; + if ((error_code < 0) || (error_code > max_err)) + error_code = 0; + pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); + } + + force_sig_fault(SIGSEGV, SEGV_CPERR, + (void __user *)uprobe_get_trap_addr(regs), tsk); +} +NOKPROBE_SYMBOL(do_control_protection); + dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) { #ifdef CONFIG_DYNAMIC_FTRACE diff --git a/include/uapi/asm-generic/siginfo.h b/include/uapi/asm-generic/siginfo.h index cb3d6c267181..693071dbe641 100644 --- a/include/uapi/asm-generic/siginfo.h +++ b/include/uapi/asm-generic/siginfo.h @@ -229,7 +229,8 @@ typedef struct siginfo { #define SEGV_ACCADI 5 /* ADI not enabled for mapped object */ #define SEGV_ADIDERR 6 /* Disrupting MCD error */ #define SEGV_ADIPERR 7 /* Precise MCD exception */ -#define NSIGSEGV 7 +#define SEGV_CPERR 8 +#define NSIGSEGV 8 /* * SIGBUS si_codes From patchwork Thu Jun 6 20:06:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980377 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C33B13AD for ; Thu, 6 Jun 2019 20:15:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 569AB287C9 for ; Thu, 6 Jun 2019 20:15:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 46DC5288DA; Thu, 6 Jun 2019 20:15:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D344287C9 for ; Thu, 6 Jun 2019 20:15:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A955F6B0293; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A45A46B0294; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 848B06B0295; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 3F6BE6B0294 for ; Thu, 6 Jun 2019 16:15:19 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id k23so2304055pgh.10 for ; Thu, 06 Jun 2019 13:15:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=cYY5bIPKktAYkfjxzWANjKvdbOk+wrI8+3g36JZ+4qg=; b=WMwgqmrrEuDweb3afoR1/w/ndV4PJYgTEhgkfhknapwhJvEbDAYm5zE3TlHsAmf66f M6Do4eEUJhvm/7vKSCZCwB4L3Xtae+K/fDdzetZSIt/ghng4Bj6QzOLmXWBvNqsEIBqM e3kE4ZLPo6ek6SsGZr82FKf0Cug64OoGc+Pr6VCnr3sFCy2zYcSETqXRVh5qYeMPpsXs mhwFfXw3+0rxl7Eo1aLBUdIIufc7mPuyj9T+M4IN0QfhS0LJJpmi5uYZumhViZIt6fC9 qNOsZHdXSL5R4CFrL4X0khabg+EnOzdLastltFKLz8JiHy0UhBADbQnCbOEGI7Oo7EmA vITg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXIW9legGAANOJgVaIsMcq/brHi0Gh8aVqxDbypMTLnTFSOphXA zih8w9+g8UI6gODAwU/Hs1OYZvbBZY+m1IbMzH9ze+WxYUzYOBnjj1ZJuE0aFne4zmlirqXjXIq tcusyX47PUqVITZyfJHofNbUO6W3ulWRLBFVYwNo+3ylsyZQ+jjcUE1wYioPpK9i1/Q== X-Received: by 2002:a17:902:2d:: with SMTP id 42mr52128747pla.34.1559852118879; Thu, 06 Jun 2019 13:15:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqyLFuL/U3j9wKvX+cK2AzMVI/wcM7cVF2CIoEZPPjr17/pGzklrvUUXNMaWxObo34nCiUyl X-Received: by 2002:a17:902:2d:: with SMTP id 42mr52128685pla.34.1559852118005; Thu, 06 Jun 2019 13:15:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852118; cv=none; d=google.com; s=arc-20160816; b=FireMYU9wXa3KyNDa36uLa7TbUcQB03LxKzP11AHoapxJHDqbHXISsgsXpl66+w6Ty ZIBmslfstlMac4plTZYlSgw8oMa/h8bEvrmeMOeb60DO121gXWgwHQEHSRT/tGjETUmN /cNNrurxZ1MMf/AOroITZTZa8GoZULQgQ2Y693IOh7s7d3GKy/osYVqBZ3Jxsvd277PR fed9o5RGrhqRExgvIHW3w4UCicuOIeDEn5SrPqf1iLNv65KAs/AFj8lSnBoGWVmxebQo 4qi3GUGAVhRHQlD1mNDdu1VhMTeiRx6jb8VBOF2Qlz1ibAax1MFuG5TAGYTxgoci9W7M 4MCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=cYY5bIPKktAYkfjxzWANjKvdbOk+wrI8+3g36JZ+4qg=; b=ioX8A1rKg3OMYaf914swmvxflFJ5rZbCZEApX15Flh1+lMJDeMCK5LwsTrltRp+oYt VHTIve/3rNHVnwYXK1pN2TCfEVCuxJtKGaDI+gu7WUU9lSB4jeyYPAn+seAgz1Fr9MP8 zd9hjJ9f8KARugDGaY8hk3i2OhYAk2tV4jtzMyj65Z1AN7GMAv/LjnDTWychzKglpDXy vGrUdwRD7td+wKF7ZTDGA3vYpuYLmMUR12seZK7ULeE0h1uDYABvvRavYOngqxgluYx+ KIfWlBmblwnf1u9tIqwESzRTivuIpFP5AL4uWoC8ClzKs3nMroyijFsl8v1YjPgqaQ4o DwGA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:17 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:17 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:16 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 07/27] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Date: Thu, 6 Jun 2019 13:06:26 -0700 Message-Id: <20190606200646.3951-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER. An application has shadow stack protection when all the following are true: (1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled, (2) The running processor supports the shadow stack, (3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack. If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection. The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 25 +++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 32 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2bbbd4d1ba31..1664918c2c1c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1915,6 +1915,31 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_INTEL_CET + select ARCH_HAS_SHSTK + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 56e748a7679f..0b2e9df48907 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -148,6 +148,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Thu Jun 6 20:06:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980379 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 739151515 for ; Thu, 6 Jun 2019 20:15:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 632EE287C9 for ; Thu, 6 Jun 2019 20:15:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 568DE288DA; Thu, 6 Jun 2019 20:15:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA1A1287C9 for ; Thu, 6 Jun 2019 20:15:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E18106B0295; Thu, 6 Jun 2019 16:15:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D79606B0296; Thu, 6 Jun 2019 16:15:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCE946B0297; Thu, 6 Jun 2019 16:15:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 7AE056B0295 for ; Thu, 6 Jun 2019 16:15:20 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id a125so2599296pfa.13 for ; Thu, 06 Jun 2019 13:15:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=gGafeXV2q0OOkveu5M5HI4+ttygaPPzx4ttepRTRb5w=; b=Ie9G+nJSHTd7nt+TdJSl7Wfnnk2jGetwXR0EhCn6YOgP6iTuAmtAakPP5/WYnlWtWV 0++DAx85iXVKLqq77KEAMdVj2z46VEvkZWIKc+0Q3RrJAo8gb+8c9rLfd8nAlec/MfzT NLlf/2ZA6QvGFWxvhCmVizPLGiFmGsQ1rRVQ/tllIxEYfZgGkVXDKTmgIJjQDmwTQ9rE Wga7SZQe6iBxoGnWluAgLiQchrFjuNBiTE/gUzuOKOqLMQejIwAxUgUI0x5MFEkIKZeh RFF4wKgwTvPQSJz5u+H8YFoUiDRGzCeMYNVuPlm06Gyb0rjPXe2j9Zm/pVopH6Gvx3sA gdCg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAU2YkIH5CVajFpttjH8HaU/GE6UXjfeXSatb5RkWpPTD0n70+oA FVA77KXhRzb1uEcG8NgIUEGkH/1Su7ICaez+udznAmqsVGsW+rOy2UNMIaULuYD+tRuZnL7KJaw t58+DF0g1RyKRDv7MWf/TPzxGYXujlVSR+3M4BZKYvDoaTjSn2jX+/azZuFBaBiMteg== X-Received: by 2002:a17:902:24d:: with SMTP id 71mr53359895plc.166.1559852120130; Thu, 06 Jun 2019 13:15:20 -0700 (PDT) X-Google-Smtp-Source: APXvYqyYcNcDguJeRjEhyJN9fUWQesiyFHaAfLvbt2GW3nEU2FQc/HN3c07ljFVIZ9k5K71WhCT5 X-Received: by 2002:a17:902:24d:: with SMTP id 71mr53359844plc.166.1559852119423; Thu, 06 Jun 2019 13:15:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852119; cv=none; d=google.com; s=arc-20160816; b=c9+EO1BjUpMBUWRp8EjRer51FB2QWj02rsP+SGBefyeMOGtisR5LApxNlORAJQJ5g3 HGtBasRIknH33eUgA2AMajtvjKo4J3p6CiDzq4XXTlixBIhXiCVEcoSx7ojTGI0EnuK1 mfhozMghNXdMIH6ZRauV9VKcZmEUdUQ8aMoB/UyMnEprQYkIwYCzpwpjHQ91nhKqPKPy GiOfYQVX+7RTz61uX3kt3EOhuW+9aYFJGkqErKoYPbKSzxUhcYCK9fvPKraZ9hxn/W9A qv+5MbAuY6imToCn+0+cmfkHbsTlFIQDs3w+l7f4K5V+xmnm61zeo+zG2GytmhM/kMqw F/fQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=gGafeXV2q0OOkveu5M5HI4+ttygaPPzx4ttepRTRb5w=; b=l0POl04ZhX86xCr9T3Az3UlXW0sm/VTVcJNO6btu2RmKRhSmWkawzGB1NdTiVYmdNs ag4t3GKkw65tEKvjPk6PcAS+m77G8rwBoeV9hnKUjxmEGHzYw2BwvVhPc1CGnJh2Y2lB wLFnDe8ApvK7hiUUHqfADjPllRQwNMqpHcVx+LMItJ69sBnboSF2XDmSntE0ohUaHxfW lSgyZfMWafAzo2Pl0jh4fB8YDA/dScTRvvWJpewsRRdbK33incWSBJcwDFREBohld56+ ygT5LTi6JQFaCdIG1X8RSUFRIbR684cIOg6r/0bUUpTcVIIQu0/V5u3+Fpl6UD6/3oA7 GmTQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:19 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:18 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:17 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 08/27] mm: Introduce VM_SHSTK for shadow stack memory Date: Thu, 6 Jun 2019 13:06:27 -0700 Message-Id: <20190606200646.3951-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP VM_SHSTK indicates a shadow stack memory area. The shadow stack is implemented only for the 64-bit kernel. Signed-off-by: Yu-cheng Yu --- fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 2 files changed, 11 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 01d4eb0e6bd1..dd8b0cd1ea36 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -661,6 +661,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "", #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif }; size_t i; diff --git a/include/linux/mm.h b/include/linux/mm.h index 0e8834ac32b7..398f1e1c35e5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -287,11 +287,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -329,6 +331,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_MPX VM_NONE #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif From patchwork Thu Jun 6 20:06:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18A361515 for ; Thu, 6 Jun 2019 20:15:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07DFB287C9 for ; Thu, 6 Jun 2019 20:15:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EFF7F288DA; Thu, 6 Jun 2019 20:15:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DBA8287C9 for ; Thu, 6 Jun 2019 20:15:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B7DA6B0296; Thu, 6 Jun 2019 16:15:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 36A5F6B0297; Thu, 6 Jun 2019 16:15:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 121796B0298; Thu, 6 Jun 2019 16:15:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id C9DD26B0296 for ; Thu, 6 Jun 2019 16:15:21 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id d7so2595403pfq.15 for ; Thu, 06 Jun 2019 13:15:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=PRMK9B749M2GnYFCAYLMoGKG7TrNJ06MBXvb3sJHopg=; b=siOlGwvuCCIL0RAeR811++B7ILom4kReJHq1g/bu0sBnUhGkDQEVwu1qYhYQYhFJlu XAxmrZhxB+W4xwv/HRCm5ho/XV90xkdjKX4uE39xqgG+tWBMyVhky2Wr7//dpTq224WT JOrqR0lDBZJccUtQsUZoF/4xpIN5n7dxSsGESdDAtRtKU6u5BQ5t8xdsYH9aCiteRKxp OhWCblKU2q2XeIMhbVxjqPHbqMHZh+UOZqnprAbHxzUry5rQvRufywmv6lJJBZ1Jyd4j L7MtPS5mJmFpCL5zFmmAihjj2e0YuCC3+dBbQnsg1Nrb+rC49rhUodDuCnqQAG8rctpn NUvQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAV4k/ZfZTbcqPmE3SpmIfQR4eG3whSKQqazUtL1XlaGx5nEOh/7 kJ0scAst9Rtqf07/NNzJXq6lVBYTtyB9tdJ5aQg6OQxagqoWS90xbySnTFzjrTsux/y13JGlyP8 ojTWIKCEYkeWsxZYTiAlqp06qicp+2tqdWEb8Bhlm6YFZ+xjJrL+ZEo5IBdX+ghuzIg== X-Received: by 2002:a62:2643:: with SMTP id m64mr53338039pfm.46.1559852121518; Thu, 06 Jun 2019 13:15:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqzme5s83GWjfWSvqpbjC/1FoO+yXlJPHYKZVuAi6RATPgGaas2qbZhV9QCMvRrjPSCCoCS+ X-Received: by 2002:a62:2643:: with SMTP id m64mr53337997pfm.46.1559852120935; Thu, 06 Jun 2019 13:15:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852120; cv=none; d=google.com; s=arc-20160816; b=Ppfb0vo3esl+fMLp2GSGHxqgI+Ew5jDWlPZ+W06vz4kl9aHeR2eXexNg3QfehhWdkH DJVPvcz1o9wOtFTsm3Qd03tfORUDM2rGmT6dTy//AhCb3HXKnOfWQL/N2LVA8SPIXnKS 0yZJAdNXJVwIzuhmcoX5W2gW3UtaxqJCAn3Bqfhi+HrkbQ/bWfdduO6QS4BpAa+nyOiu 3+kaND8qwB1qQhOCkFKI7miS0tVIp0PHji6I6q4pglY3yToSJ2EXt6Z8xpUsosUXGAuB qyWZcnapp59ZeTMXZ8aIh8KMS2CpMqrKqoGff+4HWAocSMth28Si+AEQQa2xfDhfnjYu ABEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=PRMK9B749M2GnYFCAYLMoGKG7TrNJ06MBXvb3sJHopg=; b=g/BNs+Mz7YwDIEepv6hTJLi+xYRDaT1kdgX4BmvHPgxJiHIoiUScOWhp/r21WDNTeF xsJlZjupoGYeKGKd2xMAtoCmHqvafGDMxvFbt3RKvosFQIFXrRV/4k45zsuTK/H2GX3d rUSK4TLSj6DZOjvEaRNYkFVLctfooZ9p2Jb2ltf1iW527uF03MaFC+eokkLjRJGo6Pld IS7OtYM0mvu4Me5lXSWgilUJ9fJGHjimavGR41P/x+8uSY1XKOYlt6bXVc9uBr9D1Orq SXpn/hieA0I2DWvgLQyjDd/J948/MSPQ+PrLya6UYo8EQzS+aLleWWLtBfCS7QBWwpQF EiJA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id 91si31377plh.398.2019.06.06.13.15.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:20 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:20 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:18 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 09/27] mm/mmap: Prevent Shadow Stack VMA merges Date: Thu, 6 Jun 2019 13:06:28 -0700 Message-Id: <20190606200646.3951-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To prevent function call/return spills into the next shadow stack area, do not merge shadow stack areas. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 7e8c3e8ae75f..b1a921c0de63 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1149,6 +1149,12 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + /* + * Do not merge shadow stack areas. + */ + if (vm_flags & VM_SHSTK) + return NULL; + if (prev) next = prev->vm_next; else From patchwork Thu Jun 6 20:06:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980383 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4EB8413AD for ; Thu, 6 Jun 2019 20:15:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3EB85287C9 for ; Thu, 6 Jun 2019 20:15:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 315F0288DA; Thu, 6 Jun 2019 20:15:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B8BF287C9 for ; Thu, 6 Jun 2019 20:15:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C94AC6B0297; Thu, 6 Jun 2019 16:15:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C1B756B0298; Thu, 6 Jun 2019 16:15:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABDB16B0299; Thu, 6 Jun 2019 16:15:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 64C616B0297 for ; Thu, 6 Jun 2019 16:15:24 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id i26so2582857pfo.22 for ; Thu, 06 Jun 2019 13:15:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=w6pPmJ2scNd1pZ4+36igKzH9zbFt1KQ9ZeEfcsQz6UY=; b=Y/p1xSTYOstv7xOsZnDQAgZe1RIdaY2bezmkCceGvtc64rwciX6j9wFkXfj62U9+sX OycUC3yTHkyVjmfZVAQspY+R2aPld4wJcskapseT0j08AAFSNyHQvyUrvInkdCDh4qe3 QKIJb8PW0hBMzJpylsgGOAMlSo66kIn4XYQpdqGGr6euKLiWMN96jQGOoGHgUG0txd1Y Buau7NdggyATPCDoX82t2E5PlbAlIKn647dRg753yWeXlZfaQYyp93+y3hWFKKV5y8jj tBvmwp2rRNyrC+z8GOZj47bPbxSa68wsm5txAbQP6NyNRx86n5gooMC0cLP4Too/M0X1 tvFw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUDxWbCLqgM6BvxwI+WfzAzfc1SVeHtrAf4xE2ZSA7u5mx1wPG5 yuFJ4vqUvRIadnA0PpaG4OtWtK8VndbxxAuqFtsI83h/oOPmTniAgAx7g892gJwii3TuhR7kYcl SW37ySeMfMxo9OFS9h5si+6ogka1UukHk+VeDu2ZZmb5UFdZDZ6180PepNgD41ktjuQ== X-Received: by 2002:a17:902:76c6:: with SMTP id j6mr27173546plt.263.1559852124077; Thu, 06 Jun 2019 13:15:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqzzxZMLPAP1KS6VbQ5+B3i/r1Oo/iMgZUMOVDj7PaoJ9f7z1KXAaXa6caVRox8rku8CZyEV X-Received: by 2002:a17:902:76c6:: with SMTP id j6mr27173452plt.263.1559852122894; Thu, 06 Jun 2019 13:15:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852122; cv=none; d=google.com; s=arc-20160816; b=YNkXZ2HC2MJrXPl/j4lL9KNUV3Hvh811ya51eHXTfEZq2msMQyYTc5ff76rwDmVEQW sP5mO1tsHv/nFF0eUSjjfzYXAFhoSlQIZiitcELf1vonhNNcILDUKlpVrtPsx4Y5Hx4a LLF0haWW6xJwE1mBNFSE9b3TdiajI9gCFMd5tVnzLO2f9ITizP3PPgUKuuuoXp2HhePN 3GPhKRLcOQ6qpgKwDtPYG4dzo3S4vwihm2iPv4js9StWQa5PvrqhhfGa3Ws49+I3Dpw2 F8f+y5FHB6fOlORhT4qL1+PRTk9mhZipIkob/Y25B82kzpw0roLnDpVixD0Vj5cvsVUe QHwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=w6pPmJ2scNd1pZ4+36igKzH9zbFt1KQ9ZeEfcsQz6UY=; b=FllU/7RffiQSzVntKb+c5fe3Y97oSk0bLVo/d5Xk6entl1F6hOYijs7N8MqDXdJDaG NCu7PtMu+L/IluR9zVzdzyS3trRGP+6u2PwVwdb13ynanrHMwaEMK2iCmnsz+Dvj3iue X8KOdA0MGtOiFS3ngCBIYuH2N7kR8bUwyUirYbUmBTqxNJ5J6D+k9c+zIWggbiA5Is5j LqUWryB5W52+IoGBj/aMr/i0YNkwLyTMAEoCTPJUbQFkssSYAx3pkWdzdSC8h1LJJ7kz dYJC5dlk4CAR25w6ZjDOc6QwLgNe+IeaivDBSwufiZ7hFYqP+nBDi9dX3SuTtktm4sm6 glaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:22 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:21 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:20 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 10/27] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Date: Thu, 6 Jun 2019 13:06:29 -0700 Message-Id: <20190606200646.3951-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Before introducing _PAGE_DIRTY_SW for non-hardware, memory management purposes in the next patch, rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5e0509b41986..5008dd2817e2 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -332,7 +332,7 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) @@ -406,7 +406,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -460,7 +460,7 @@ static inline pud_t pud_wrprotect(pud_t pud) static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index d6ff0bbdb394..cebee656a250 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -15,7 +15,7 @@ #define _PAGE_BIT_PWT 3 /* page write through */ #define _PAGE_BIT_PCD 4 /* page cache disabled */ #define _PAGE_BIT_ACCESSED 5 /* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6 /* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6 /* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7 /* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7 /* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8 /* Global TLB entry PPro+ */ @@ -45,7 +45,7 @@ #define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT) #define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD) #define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW) #define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE) #define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL) #define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1) @@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3) #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED) #else #define _PAGE_KNL_ERRATUM_MASK 0 #endif @@ -112,9 +112,9 @@ #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) #define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * Set of bits not changed in pte_modify. The pte's @@ -123,7 +123,7 @@ * pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED) #define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL) #define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX) #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) @@ -187,7 +188,7 @@ enum page_cache_mode { #define _PAGE_ENC (_AT(pteval_t, sme_me_mask)) #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC) #define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER) #define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index 11eda21eb697..e7665a4767b3 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b93e36ddee5e..9b3b229b5ef0 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3364,7 +3364,7 @@ static int init_rmode_identity_map(struct kvm *kvm) /* Set up identity-mapping pagetable for EPT in real mode */ for (i = 0; i < PT32_ENT_PER_PAGE; i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r = kvm_write_guest_page(kvm, identity_map_pfn, &tmp, i * sizeof(tmp), sizeof(tmp)); if (r < 0) From patchwork Thu Jun 6 20:06:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D40511515 for ; Thu, 6 Jun 2019 20:15:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2A21287C9 for ; Thu, 6 Jun 2019 20:15:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6330288DA; Thu, 6 Jun 2019 20:15:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1D81287C9 for ; Thu, 6 Jun 2019 20:15:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B1436B029F; Thu, 6 Jun 2019 16:15:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 230FD6B02A1; Thu, 6 Jun 2019 16:15:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05BA66B02A2; Thu, 6 Jun 2019 16:15:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id B0B086B029F for ; Thu, 6 Jun 2019 16:15:25 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id u1so2324578pgh.3 for ; Thu, 06 Jun 2019 13:15:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Y9Zc4N6NG2Z4vNfKi0LlVFvk0a5eJ1nr33xkyZozgsQ=; b=RG3iJQFlTDvm9fOHerEDatPdb53See+7IYz7KcUZKKDbGTHEWbXLfkZmrkOtVW8A5x bBOErfktxBi1LzXP98wCniXPXZF5HaRz5QjmMu8a6rCDb+Y5d51L+CL8atI1+KDhjhcD uuJXJC3Zy8V/hdySUjcsx4CdObhF8nDxz+al6bxG5/2zhIC6wUpHJ5zqCKsh1X/wll6n alka5xG9xFmeNuROGrxT3f6uE4OdRdHW/FkYicVaDKgIoflOfimSRoM6wUkQ+aWNrkQs g2g6Qf/f//ZWm1zlgFexQfM6nAPCY60lFFcbV4CG44v8l5vpyvQoNl995Z3MiY2szSUF 7m2Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVsdClPjw535PcA3zaFrA+Tus+fzZF7RwcD7k6ulaWbTi1hbUZK fLs2NSX5T7OZ+UOrv3LpPmBJlnw096XUQdU1Q4HhKx+jhOmVo2LkeMXRB2oR1JdvfVpZLyRdFzt oIM7sMtMdybBApg/EfSuEIWYcM9MqeO097MH18A+/6o+m8GunjLAlKHvNoF89knFPYQ== X-Received: by 2002:a17:902:9a42:: with SMTP id x2mr35309714plv.106.1559852125339; Thu, 06 Jun 2019 13:15:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqxYTDYdf+Pua8kO2qOkAS5y13zUAUtc53hSqivrapO70oj5hKbisGuJ117SkrGHOMiPnj8u X-Received: by 2002:a17:902:9a42:: with SMTP id x2mr35309596plv.106.1559852123741; Thu, 06 Jun 2019 13:15:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852123; cv=none; d=google.com; s=arc-20160816; b=nODq81is8oEXPDURsgD5vpPqHNxdgGQYNMKFP12Htp1vVeEvdwBub5AUegjuZqgQ6Q Q3bxjNcrxFRObOSqjrbAjRyWDgH0+UCgr+Giv/OfLqgz5LPWOln7In7kjsXci1wivldb lsdgIkeGY/+aPYsbVoBtO5v/HVyk+U4nZHfUbmFns/Q9C/nrFMwD0JBvbQ4ndCb3ZCje SR/454IapE+5xcMRdlojKSbiEiLkVbRcbOeZTofxclpPGCcQvoCeJ0GOTiDRyIhyOgRI XmZM9cz/DhSHpg1m6+Or+eM6cv2B1trELD1SJgRNeXbKsdyUjypy1CrPdPBpvOVuqbc6 7sng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Y9Zc4N6NG2Z4vNfKi0LlVFvk0a5eJ1nr33xkyZozgsQ=; b=hM13pm1aRiBVVK80iYYHsUwdo1XUn3I7m3VkpQDeZ3l3YBr5iwo5sgCSquWiO6WO4/ P1tUUOEpdrW/UNLMXNnWHOBGiBaIPnM3MtdEO2FOY8tv8r0GdyJ5vcTz9R9+NNR5Mw58 SyhsvPWPrZxCLlZy44gTKR3CCGN8CGwZr6PRniABErIYCBgfjNZulKXvn4bJE6wQ0461 dhfwxewUTxNuQD/uVeyEc4bKYi44v5jpIoEcDBzcworAuWpYvXlQcbxpQQqAb/ajTl6n TAyV/nBlsxDoNb9mE7D/KlRT0liwkqtV+B0t4LiIiTWz2C5YhRFlp+O/+MLq7Croefy0 pWwQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:23 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:22 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:21 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 11/27] x86/mm: Introduce _PAGE_DIRTY_SW Date: Thu, 6 Jun 2019 13:06:30 -0700 Message-Id: <20190606200646.3951-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A RO and dirty PTE exists in the following cases: (a) A page is modified and then shared with a fork()'ed child; (b) A R/O page that has been COW'ed; (c) A SHSTK page. The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings: Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW) Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs. When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 21 ++++- 2 files changed, 128 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5008dd2817e2..0a3ce0a94d73 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,9 +120,9 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; } @@ -159,9 +159,9 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } static inline int pmd_young(pmd_t pmd) @@ -169,9 +169,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -310,9 +310,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + if (pte_flags(pte) & from) + pte = pte_set_flags(pte_clear_flags(pte, from), to); + return pte; +} +#else +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + return pte; +} +#endif + static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -322,6 +336,7 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pte_clear_flags(pte, _PAGE_RW); } @@ -332,9 +347,24 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { + pteval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + pte_write(pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + pte = pte_clear_flags(pte, _PAGE_DIRTY_SW); return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw(pte_t pte) +{ + return pte_flags(pte) & _PAGE_DIRTY_HW; +} +#endif + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -342,6 +372,7 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pte_set_flags(pte, _PAGE_RW); } @@ -389,6 +420,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + if (pmd_flags(pmd) & from) + pmd = pmd_set_flags(pmd_clear_flags(pmd, from), to); + return pmd; +} +#else +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + return pmd; +} +#endif + static inline pmd_t pmd_mkold(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_ACCESSED); @@ -396,19 +441,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pmd_clear_flags(pmd, _PAGE_RW); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { + pmdval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pmd_flags(pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY_SW); return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_DIRTY_HW; +} +#endif + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -426,6 +488,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pmd_set_flags(pmd, _PAGE_RW); } @@ -443,6 +506,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + if (pud_flags(pud) & from) + pud = pud_set_flags(pud_clear_flags(pud, from), to); + return pud; +} +#else +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + return pud; +} +#endif + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); @@ -450,17 +527,22 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pud_clear_flags(pud, _PAGE_RW); } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); + pudval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pud_flags(pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -480,6 +562,7 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pud_set_flags(pud, _PAGE_RW); } @@ -611,19 +694,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); + if ((pte_write(pte) && !(pgprot_val(newprot) & _PAGE_RW))) + return pte_move_flags(__pte(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); return __pte(val); } -static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) -{ - pmdval_t val = pmd_val(pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; - val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); -} - /* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modify static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) @@ -1178,6 +1254,19 @@ static inline int pmd_write(pmd_t pmd) return pmd_flags(pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + pmdval_t val = pmd_val(pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); + if ((pmd_write(pmd) && !(pgprot_val(newprot) & _PAGE_RW))) + return pmd_move_flags(__pmd(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); + return __pmd(val); +} + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index cebee656a250..1f04551c55f6 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,7 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57 /* available for programmer */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -34,6 +35,7 @@ #define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ @@ -109,6 +111,21 @@ #define _PAGE_DEVMAP (_AT(pteval_t, 0)) #endif +/* + * _PAGE_DIRTY_HW: set by the processor when a page is written. + * _PAGE_DIRTY_SW: a spare bit tracking a written, but now R/O page. + * [R/W + _PAGE_DIRTY_HW] <-> [R/O + _PAGE_DIRTY_SW]. + * _PAGE_SOFT_DIRTY: a spare bit used to track written pages since a time point + * set by the system admin; see Documentation/admin-guide/mm/soft-dirty.rst. + */ +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ @@ -122,9 +139,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) /* From patchwork Thu Jun 6 20:06:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980387 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63C3013AD for ; Thu, 6 Jun 2019 20:15:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5215328A6D for ; Thu, 6 Jun 2019 20:15:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 45E1028AA5; Thu, 6 Jun 2019 20:15:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5A9E28A6D for ; Thu, 6 Jun 2019 20:15:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6EAC96B02A0; Thu, 6 Jun 2019 16:15:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 69E5E6B02A1; Thu, 6 Jun 2019 16:15:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CDD26B02A3; Thu, 6 Jun 2019 16:15:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id EDB4D6B02A0 for ; Thu, 6 Jun 2019 16:15:25 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id g9so2285251pgd.17 for ; Thu, 06 Jun 2019 13:15:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=+sTJWF6AyOPSr6M0VfrE46RFU8heKaSNwIJptODPH8I=; b=Obc1/GH+Yv5nSsrMkJ0QAqjdsNeeAYMP21ZiQaY2PaGZ/fbaKZB9S7XgeQcGJ76bu1 RwFdRtD+JvnVL+6LJTWknak7IVBDIdEpyobRUK15SNdPmvBp3k5Mfld5RY/JQOHGl9fW uMxYnFG0KB5EU78c5lLG+TQaUZ+MZRnGZPeaIpfTH/89OL6BUzaogWDyb+kbtKjNl7vk P2V8DO49qrrwiPW+4mEPWrBCrxE/B6Cka6IHxPfOnMa3wzaWmMKrrYGQFhHNoeeq3ep0 WGxQsgJgbhOsM+IEsPKUF2iZgPDJggNQp2FrHAXyNlx6zmM3K/727FD8JDL8C1EvEFoo SKiw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVQ2eN31qa9cg1NLLT9EBfGGFHe0yHQdlDZHyIdhIJJ2GI0AxQj H0MnbRSKgNNkBSJ92i7Ee2F64tXvsFgWbtzqTVumlUbK0B323MZjj8xdZJwbAPIU5Ax23uUODjn yttGXgHqGZIb9hSmhQUNbZRh3fbvQQDGWAvvpISuSLOKY59TqcnhkZdiTqFfeT8XGBg== X-Received: by 2002:a17:90a:2e89:: with SMTP id r9mr1616856pjd.117.1559852125651; Thu, 06 Jun 2019 13:15:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqxUHICzcUzGZOuuqv3C6gE/Cz/BhCfttS4fszVZS620aSpNTdDhCowTBWQP0sW8gn2UFTrv X-Received: by 2002:a17:90a:2e89:: with SMTP id r9mr1616794pjd.117.1559852124700; Thu, 06 Jun 2019 13:15:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852124; cv=none; d=google.com; s=arc-20160816; b=0x8KbWjL2b2D7JGqqYAzvOueZKwBZ2iTv3zgXxVsd4rWJNfso3qtX8C6q13Gmt2Pqo CmoL3PMW1seGPLC9O9/nW1o5YwiqGX/PSutDrmU4tf42NSXuPVLAUWDjEJHOMSwMShaT NYYVMjtXlnkjCX/lZTPa/aDpeK7e6N+lgy48gCLWH5vN77MHpFkxKqU2WZgaC1t5ztKt Aa/3aoWIYegFoIpvkUUAQX/lmSlX2+idRDuOBYPxEPz+ZlX0VSTCrt55slqY0i+uUrB0 fnsThCjGAdmn+tN3goWdEiipqnXMTldQ5BhvrHC68zWfStjkzEBd5tXhNIt1PM5k/9Di DwPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=+sTJWF6AyOPSr6M0VfrE46RFU8heKaSNwIJptODPH8I=; b=ZQUPfAAkQ0X0ZKGeRI58JU/ky7mcYz2mY4xUgBnZC7g3vSdwNpNwbCchvl15gTIvUD nC8zWF1nlhopnaZvt2cx7yy7c+bLb8YPdtn3pGhAuQs4+7gJP7hEvydfq9hnB8y7Xqp6 9GOGan+QmCAJH6vbaTTaaEcCxVqwvIBsGeqwqeztpTHNbgSZL0mfCLIs7fQIBjy5raH6 g8iUmYO2JQF3NOXzH+VhrrXV2nTUTUMgcul7iKdEObiFzgjwDcpX5d9omu1ZK2BcGvBD lRJowc8Gb3ngopANmvUfNsM1TC9XHq6GaxT2+JDHb/tWXayDeHiVvFYj1XTvUHuVo0qv FQvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:24 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:24 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:22 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 12/27] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Date: Thu, 6 Jun 2019 13:06:31 -0700 Message-Id: <20190606200646.3951-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Update _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry(). In order to support Control-flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW. Signed-off-by: Yu-cheng Yu --- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 244ad1729764..44f35880d771 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1185,7 +1185,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, } /* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse(se); ops->clear_ips(se); From patchwork Thu Jun 6 20:06:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55C6813AD for ; Thu, 6 Jun 2019 20:15:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 44E1928A6D for ; Thu, 6 Jun 2019 20:15:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 38C2928AA5; Thu, 6 Jun 2019 20:15:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A030628A6D for ; Thu, 6 Jun 2019 20:15:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D09096B02A1; Thu, 6 Jun 2019 16:15:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C91C76B02A3; Thu, 6 Jun 2019 16:15:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0C716B02A5; Thu, 6 Jun 2019 16:15:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 718C06B02A1 for ; Thu, 6 Jun 2019 16:15:27 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id v62so2324385pgb.0 for ; Thu, 06 Jun 2019 13:15:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=oj8Un5QmH3g727TAPRPmQHhqB9MuswGdM4DJNrA223w=; b=ctn1nkWspR6ZFzP2/Rj3p2eEGo9+mwzvpjE0CiRusYf723ormWmDny0Nzd0+1IANS+ p0ZNV5hZAzOuICkUB3Rpu/xe0OSHd1rCtBGgHSn0I5jMXw+p4T6BMGcGyg8svHMR1dxR xmd+vJLsedfu65T4UHCM0K5+N/DseHc+y3AXLmLTEnSAgxbbl3H/c7GwxuFVj1Vn2kVW m95hblaGlScLeyEjqHRpVqKDIIc/BEx5/niWc92cM3QI3r7QP7qtqIoS+PI9YGLB1OH8 jPEjEFcAfvOlhELKNX2rfpFSXZm3f1a7qcNfa+dye6jH6FWt/tBpwp8ZNoQkK7W04R4m s2Kw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAW2RtfCpd5eO91lwEJX2BIfjLPYp2AfhnQ8DdL7HwK8lriFYfFe mrRtS7mnwArAxZRsNo9LnmuWNZntm5CGpqd3SpqKoA34KohaMno661/OOMr4EGVXuKpfJs9XYH6 0GEbi/Cn42+nmgFWByobJrulj+Wj5q3YAMvjczbPLFf41oZJcoIEdPMuawg9T85FnDA== X-Received: by 2002:a17:90a:730b:: with SMTP id m11mr1616599pjk.89.1559852127114; Thu, 06 Jun 2019 13:15:27 -0700 (PDT) X-Google-Smtp-Source: APXvYqzY1Rc9dY5qV2mLtSI6pAdTXlvh/5FIVVPvK6lEuEPYD+5Woa+32najWJMsaBqvEwyggaLv X-Received: by 2002:a17:90a:730b:: with SMTP id m11mr1616543pjk.89.1559852126218; Thu, 06 Jun 2019 13:15:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852126; cv=none; d=google.com; s=arc-20160816; b=R84NgnYhvLN7A11C3rrVcU3ZTahMw1NFdYJ3ib+RQDRq9Ec65jQTIC9dhkAsBmvYek v9Teo9zCINFRa/6OipD9P4r02Fwlsbf3E3Kb2jdRhrmyUcWamrcwfPQaVf+fIZuiP6dT HY7S1mAmBGPOPEF3j05Pe12gELVcVcCo37VG1jy241oy6E0e4eOqU1J/wrkOzMXvtsdy wdFpzPTP3kh8ueLFILj3EHXR8/+Kc55OUBLwK2G+xyxSvrBR0d/sHR4na2SvaZSoLQhk cnUYWKbXafQUn6XDS50jDA4/CZzQbLScJZHnBVHaTZ1DRpLr/qXdsg/n2uPGozVjTYyn qomw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=oj8Un5QmH3g727TAPRPmQHhqB9MuswGdM4DJNrA223w=; b=d/bC62drPQikpFO5Qn4bs9nyFO+41RRewbt3QJFMMjb2268sbnp37h1ZZj/zKaR+tg 9gR0q3NBVz3n3LByno71F3J9SZKUNs/7sZpJuf1YwQqAvOWGlupWgRsNkaooqFTgqj5b mnGy+pbshtGu0IIpd6y3XuguPONhW8Euye4MJiX7ic50sEqkE/UpJ65Nq4TFXJeJagQa CH5IhBqkygyYiIHwDdn5mVileatZsnQlJIAUdOPZUMeWyc20ORUCsBJ5+lRNYTlnJr71 uA5ViriYjREXP+V9Vix0e2ieUW6g4p2jQJ8e8OdlqRgqh69JK3Z4QRwxfnYv7u58vY4K Ioew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:26 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:25 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:24 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 13/27] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Date: Thu, 6 Jun 2019 13:06:32 -0700 Message-Id: <20190606200646.3951-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When Shadow Stack is enabled, the [R/O + PAGE_DIRTY_HW] setting is reserved only for the Shadow Stack. Non-Shadow Stack R/O PTEs use [R/O + PAGE_DIRTY_SW]. When a PTE goes from [R/W + PAGE_DIRTY_HW] to [R/O + PAGE_DIRTY_SW], it could become a transient Shadow Stack PTE in two cases. The first case is that some processors can start a write but end up seeing a read-only PTE by the time they get to the Dirty bit, creating a transient Shadow Stack PTE. However, this will not occur on processors supporting Shadow Stack therefore we don't need a TLB flush here. The second case is that when the software, without atomic, tests & replaces PAGE_DIRTY_HW with PAGE_DIRTY_SW, a transient Shadow Stack PTE can exist. This is prevented with cmpxchg. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 58 ++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 0a3ce0a94d73..2dd079080be2 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1222,7 +1222,36 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pte_t new_pte, pte = READ_ONCE(*ptep); + + /* + * Some processors can start a write, but end up + * seeing a read-only PTE by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PTE and flush the + * TLB would be needed. + * + * When changing a writable PTE to read-only and + * if the PTE has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PTE is + * not a valid Shadow Stack PTE. + */ + do { + new_pte = pte_wrprotect(pte); + new_pte.pte |= (new_pte.pte & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pte.pte &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(ptep, &pte, new_pte)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +#endif } #define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) @@ -1285,7 +1314,36 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pmd_t new_pmd, pmd = READ_ONCE(*pmdp); + + /* + * Some processors can start a write, but end up + * seeing a read-only PMD by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PMD which looks like a Shadow Stack PMD. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PMD and flush the + * TLB would be needed. + * + * When changing a writable PMD to read-only and + * if the PMD has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PMD is + * not a valid Shadow Stack PMD. + */ + do { + new_pmd = pmd_wrprotect(pmd); + new_pmd.pmd |= (new_pmd.pmd & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pmd.pmd &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(pmdp, &pmd, new_pmd)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); +#endif } #define pud_write pud_write From patchwork Thu Jun 6 20:06:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 877E113AD for ; Thu, 6 Jun 2019 20:15:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7663828A6D for ; Thu, 6 Jun 2019 20:15:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A46828AA5; Thu, 6 Jun 2019 20:15:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E8AFE28A6D for ; Thu, 6 Jun 2019 20:15:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 342326B02A3; Thu, 6 Jun 2019 16:15:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A3186B02A6; Thu, 6 Jun 2019 16:15:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0818A6B02A7; Thu, 6 Jun 2019 16:15:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id B7E2D6B02A3 for ; Thu, 6 Jun 2019 16:15:28 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id w31so2274149pgk.23 for ; Thu, 06 Jun 2019 13:15:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mJv/WQK1WOmy8hTCtd8EzUl4KsxUY16rQBl3AY6S500=; b=J4bc4wuRN9ilGgJ8TnUtNTzLTBEVj6ksbCsvDOHojDiCin5k+cJLeYhlPT8lspTluh AfrcYm2mIHF+z8m9NsSfLykx6vXCRP+RN6s0u19IjCL6mTGJqb5uBtrslEPME4Jhnjch gDR8N4iy2RWSFqvfnT3PupRgcsrrtoeJe1lKo4uBsE8JV5JmaniUXkK+IU84j/AdGClE qB4Bh65Yr32L+qO30AXhKT45id977ZUuER4+c3Hz79uwrTLonq8+SKOAQymGEsvnfp0x bbBMXv/XttigfT0Ki5Y3nagVneZCken/7ommoUFOGw/6zYOQO2yWEE00sF+f4qrfXfxH /0yg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXX5ixZnDec0lfTD1V2kyt0WEn4Mxe6AdFj8lUKmO54+b3Yq0M0 14c72rj0iG8F82uRpGC6qzOYPeKOxpNGwa3e8wgB8WL8G/jfIzvXc6Y9lk7f3p6gUE2aZ1mQHW8 QekYTNPbRW7NP4HcDNo+xkdnJiH1Hp0lVA/A2x5x8Kn7dfwf0pQLqWNwjfJXmm8IDvQ== X-Received: by 2002:a17:902:e105:: with SMTP id cc5mr31380500plb.320.1559852128421; Thu, 06 Jun 2019 13:15:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqzGexB0SFKiRWdSEg+M1Vzwi0g7gV/EblNgCuefSXpg+UGDyQZfDIt78VzlVXxqPnkKNkpc X-Received: by 2002:a17:902:e105:: with SMTP id cc5mr31380427plb.320.1559852127343; Thu, 06 Jun 2019 13:15:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852127; cv=none; d=google.com; s=arc-20160816; b=iVAC1WEWRkSLedJbLWPXc/rfe0NzkOCrHFx9ch7SGe3EbBGUG9ffoeXBFVNzHdYFcv kfCdYOsE14kAuTXx6Ct6C2oGg/vNL9V62ayMRi6B2L0yY5ldSDRk6bJucvimDgNbIkOt Vlby3xeOF61+oWJ6rw6RalPkRqrlb7hrAT7RgAf1Lsh856/LBwnSOR7Rf7Cz9bh3TxGo UYMrXDkM0CCq4jOTtmsa0YA2fAa9gC4SoHfX/Qu3ya+fsX/9zuMuADpGNgaj50sDXkO3 3CoPj048kaea7LmxhpZf7JUdzmEiVPvKULXAKZaOZrGBRgP8JCW0K6joTAyzzIpgA0XY nzNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=mJv/WQK1WOmy8hTCtd8EzUl4KsxUY16rQBl3AY6S500=; b=IA09WHrf+pSev3kVDpPIkIcrdArXlutmvxzXAQnWHhgNSL6/s03PySRH5hvvbdSoOT pfLABIfOzYVbYnOmOO0ZKkwRL+xeJNycJH1vFvW7MMrhk2qNc6i7IvO6jxW4HhyxhkQs BD/rKHttUCiyrYrZ3l0VE1WsrZevLPjWiabcNyvr8/f6PgRCCAx13dBdOji9WIctjrpr 9G39DipKZK1LqZxTK1v5tGMBPg9onCTdaEbE2tdhvyvKBu6dwT64RfiTZYKH/7Fr4gA+ 6cT537lS1vMQu45PGxDZ8n6+d3fsCoOWwDfZsswmGpkxYO/Sg3LO6o0qH4vWhdAwkbYb Lnew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:27 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:26 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:25 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 14/27] x86/mm: Shadow stack page fault error checking Date: Thu, 6 Jun 2019 13:06:33 -0700 Message-Id: <20190606200646.3951-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set. In access_error(), verify a shadow stack page fault is within a shadow stack memory area. It is always an error otherwise. For a valid shadow stack access, set FAULT_FLAG_WRITE to effect copy-on-write. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 5906a22796b6..292fb3a1b340 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -166,6 +166,7 @@ enum { * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code { X86_PF_PROT = 1 << 0, @@ -174,5 +175,6 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, }; #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 46df4c6aae46..59f4f66e4f2e 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1205,6 +1205,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Verify X86_PF_SHSTK is within a shadow stack VMA. + * It is always an error if there is a shadow stack + * fault outside a shadow stack VMA. + */ + if (error_code & X86_PF_SHSTK) { + if (!(vma->vm_flags & VM_SHSTK)) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ if (unlikely(!(vma->vm_flags & VM_WRITE))) @@ -1362,6 +1373,13 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * If the fault is caused by a shadow stack access, + * i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set + * FAULT_FLAG_WRITE to effect copy-on-write. + */ + if (hw_error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (hw_error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (hw_error_code & X86_PF_INSTR) From patchwork Thu Jun 6 20:06:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86A9B6C5 for ; Thu, 6 Jun 2019 20:15:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7532E28A6D for ; Thu, 6 Jun 2019 20:15:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 67FEF28AA5; Thu, 6 Jun 2019 20:15:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A61E28A6D for ; Thu, 6 Jun 2019 20:15:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DA0E6B02A6; Thu, 6 Jun 2019 16:15:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 58ACD6B02A8; Thu, 6 Jun 2019 16:15:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 409516B02A9; Thu, 6 Jun 2019 16:15:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id ED9326B02A6 for ; Thu, 6 Jun 2019 16:15:29 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id 30so2292985pgk.16 for ; Thu, 06 Jun 2019 13:15:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=kOQHRtvRp7Krjq9aXSeGB8Dyu2+i8rD/nMOsAT9UzjM=; b=ZYFSVgVA1D0C1yG242aoElI9zhCvpzYxEtyKbUmr7E3sI1+AzUCvXP4LIP6NW7Di8e 7eMp89KZhJaqUwX3mgJ6iQah5SKnQvGzpBT/1GT1grK/hWng0pukd4SzRKlsJHhOvYhl P1oobPBllZel9d3QpSAaYIwGn3WHwnC/UYn8sg2P0JyxnSMHgmSWzCvEa2mTTn7zaTcj ldSL/6t3alCdD2x1oqZi+ojJpnqE/q4btq1pW2dE0+SDvcTD/O7O62W3szM/tcGUamgg muYXOGAUFFmext4c6KH/odxQja+p/nbjC3BsJNszEv2WMYYuQ6y27ilGnUc1q5eCL9c1 Qi5w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAU63hLsijliF1XMHPHvdQNdJTX1E/WzYKMxhHQQtrEkpsQGd++O LlMwyv8XzXiLC98insjmlwZ7qSxQfbKCzDOX6Hx909vIz+8Rl+SkaQe6U1SGvwE1jHHlCZ9i509 eWPAvgRPaa69TZQa/9TTRNEvPlzczBvKxLUZWoVnSkWI+jK56tqaWEWAA4F7+RWb3MA== X-Received: by 2002:a63:c5:: with SMTP id 188mr328991pga.108.1559852129530; Thu, 06 Jun 2019 13:15:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqzn6wdatoq9PlympjMiikedR1JRfe7OPuEOUR2TdIBblSclMa1aeN92znOPllW2uLZp076S X-Received: by 2002:a63:c5:: with SMTP id 188mr328944pga.108.1559852128707; Thu, 06 Jun 2019 13:15:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852128; cv=none; d=google.com; s=arc-20160816; b=0MNsDImkkdaKM4LNs5JYm3uiZDoYaLk77FCwaoeKFZOW+xFj5rGfa4tVG+B862yVXf QROVocazay8xmJal7MqbrIhYnuTJQbpkBmiHROQDAhVolieUsRkbucEUDWB7KAMgi/YX eR/SNLa6zn+oWTbmp8TiCj8rzMJMSW1oWD4ERsgkIKHi+uxsInOw4rHl3fbvY2Qaa5rk W2etnCn9MyvX55vSpDy9UWODFinoZsPCcQm4VY6c6dNPiApNsZi1IeBizOUPEnNTEwx+ 92lDekz0qrTFQ+/jtNG5J2ALgn2cGAhpQIRykojvjoZz1Yne9B2/aDduADQSZfcrH6zF OEiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=kOQHRtvRp7Krjq9aXSeGB8Dyu2+i8rD/nMOsAT9UzjM=; b=j4pbYfpeKuWMB8LvGaqfWgHw9GqQqSD4Vdbkiz6bBxTEMeOEJrC4rGHVdcUJypgyou 7Pkdrot0Kqsyb2pgCkSS7gXVv9x8dl5w2rD76gIKhW/Kq3QFwug02lpx6JRW718Y5E7W b5TZJ6NzNpn7CQZ7kDuZzpDAR5qkLCrfG6vgkiVNL4tBtdxAAxZ7x+hnuwt69260d56o 9/fNHPhdYsm9S9mOAZajKedSymMA6wyhFoEh1xnpCgMJYVrHHmCevMxMhg/6grOGKO8H nd8wPRSX7tCyZpSi9tx37mu5WEl+NS4nW8au+06Ce2SrSpq7MTG67zm7UfaKfsP3zNf0 VQ+w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:28 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:28 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:27 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 15/27] mm: Handle shadow stack page fault Date: Thu, 6 Jun 2019 13:06:34 -0700 Message-Id: <20190606200646.3951-16-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a task does fork(), its shadow stack (SHSTK) must be duplicated for the child. This patch implements a flow similar to copy-on-write of an anonymous page, but for SHSTK. A SHSTK PTE must be RO and dirty. This dirty bit requirement is used to effect the copying. In copy_one_pte(), clear the dirty bit from a SHSTK PTE to cause a page fault upon the next SHSTK access. At that time, fix the PTE and copy/re-use the page. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 15 +++++++++++++++ include/asm-generic/pgtable.h | 8 ++++++++ mm/memory.c | 7 ++++++- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 1f67b1e15bf6..c2d754a780b3 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -891,3 +891,18 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_mkdirty_shstk(pte); + else + return pte; +} + +inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) +{ + return (vm_flags & VM_SHSTK); +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 75d9d68a6de7..ffcc0be7cadc 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1188,4 +1188,12 @@ static inline bool arch_has_pfn_modify_check(void) #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +#define pte_set_vma_features(pte, vma) pte +#define arch_copy_pte_mapping(vma_flags) false +#else +pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +bool arch_copy_pte_mapping(vm_flags_t vm_flags); +#endif + #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/memory.c b/mm/memory.c index ddf20bd0c317..51c97294f00f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -777,7 +777,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * If it's a COW mapping, write protect it both * in the parent and the child */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { + if ((is_cow_mapping(vm_flags) && pte_write(pte)) || + arch_copy_pte_mapping(vm_flags)) { ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -2312,6 +2313,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2387,6 +2389,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -2910,6 +2913,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte = mk_pte(page, vma->vm_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); + pte = pte_set_vma_features(pte, vma); vmf->flags &= ~FAULT_FLAG_WRITE; ret |= VM_FAULT_WRITE; exclusive = RMAP_EXCLUSIVE; @@ -3052,6 +3056,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_set_vma_features(entry, vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); From patchwork Thu Jun 6 20:06:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980395 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C13541515 for ; Thu, 6 Jun 2019 20:15:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B00BD28A6D for ; Thu, 6 Jun 2019 20:15:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A037428AA5; Thu, 6 Jun 2019 20:15:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A59B28A6D for ; Thu, 6 Jun 2019 20:15:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84AE36B02A8; Thu, 6 Jun 2019 16:15:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7D6C76B02AB; Thu, 6 Jun 2019 16:15:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6768B6B02A8; Thu, 6 Jun 2019 16:15:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 0FB896B02A8 for ; Thu, 6 Jun 2019 16:15:31 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id d19so2184353pls.1 for ; Thu, 06 Jun 2019 13:15:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=wTis/tBUSVnfnxqsMN3VsGqgVPF5X8eJdrNigljA9pg=; b=AApvlGjcpSK5lEgGaBgUtYIHv2U1zaGY+LvU3suRTb9Tw2KJRpGSeJK7qQEIep/kkk oD0Ur6UwWrr0P5zkLzbwDf11XoKDqjT0dmPh3V+GlcbeIrHKnhl2ICH1/GvWbbxc1Yoo G0JZu4nQxPASpTpVjV195o8G5DSnWLMlxn5URXj87TVTYB+0SnFqR7ktJWmo4BCdKOzK Jfi0Ud5KtHVcsgukSOCrDgYQhxcDvEYyGTAco7/aQac6BWi4grv8GiAmvV/p2VRwGQ+O VuH9S8u3oWW0CcCEJstr49WKxNmFIH1IcgX4cOsSHRBKuVkSY3ZbwTZJaCSHGOntDTzD 6Hfw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUVvD7Jj3Gi9IeQvTicKXwVGd+68ooZNzy9v1C2duAjy1Uu9oV9 lmyoPoONhCt289MjrAgLiyFIn3wMj31ewDLYVr8bUHikdvlmUp0ZwYnL7oTn/gRg9uMfmkGaFNZ OsDcyIXdWzVYv8Cu6ArB/9FIqAHlDZ4jGhwFbDeytjGcquJpyBv60+eCar69N7scPUQ== X-Received: by 2002:a17:90a:a608:: with SMTP id c8mr1569319pjq.37.1559852130730; Thu, 06 Jun 2019 13:15:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqwhMY+wvibVgWXRC6e7tqsqNZQ7xxK4HNEaGIKx+5AFRqJP3Kuu6EdDlLLnMlp0APQ2808T X-Received: by 2002:a17:90a:a608:: with SMTP id c8mr1569260pjq.37.1559852130000; Thu, 06 Jun 2019 13:15:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852129; cv=none; d=google.com; s=arc-20160816; b=GXoATEyjA7eE0T8Mo9c5kZj1g2DXSmNIG2GT2mKOlvWlFS4ta7PtrFBw91CvyRT9vP 47DWim1k/x9PG5qr+LMG+DOI7ZxpRztii8FVm1ezPjCzEJD3hfdyqRjYqChEtFK9Vd5V oX/Sr0JGrsP2ns3nW/VOSwcRCNcit5ARKYKRvBesqRHPtONyipTKHIomhkwgYNbcFNZM eu+JXM626DioFlg7sb3tPrGcK5uznxV5CslwNlwD9gLQlMu6OwC6f4QK3d7HBDTh0seg CLBaFrOi6edfyT/5MuNXek731jnvv8S4Y6p4/hLjMRft6cbaNg/mPWaXSgaIZH+QQBRm pueg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=wTis/tBUSVnfnxqsMN3VsGqgVPF5X8eJdrNigljA9pg=; b=N9jErH1ezpIm63pgkS8GrAqPK8bhJtQtw4YXtXiRax0FuSD3yseruQ53eGQgCT6xMn dKNcdDCKt6+7rY6WyU/tkezCXLTJG35RmtgYcUGnh03b447/UFWWp/K8CYI2YiWMuCM2 AdnG8giojqlBJbfvUXRbfujcQo2NekVCZKe8RyrgKMpKiGEUa5+GkmBTLgWUke6LZzNY TgTX+PvDMSyg2RdzKMWLIKXJ7AZggBlGqz0lYDMoo5IFiPKf26RgTsXqR/oL0wyb9oKN ZlfjaUKXpcs6FXT1aUu7eIjboxEzIZ0BfM8SU1v3yHThCx0JzQ6xIiapiwpYHMe0lKZr v/Kg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:29 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:29 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:28 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 16/27] mm: Handle THP/HugeTLB shadow stack page fault Date: Thu, 6 Jun 2019 13:06:35 -0700 Message-Id: <20190606200646.3951-17-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch implements THP shadow stack (SHSTK) copying in the same way as in the previous patch for regular PTE. In copy_huge_pmd(), clear the dirty bit from the PMD to cause a page fault upon the next SHSTK access to the PMD. At that time, fix the PMD and copy/re-use the page. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 2 ++ mm/huge_memory.c | 4 ++++ 3 files changed, 14 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index c2d754a780b3..8ff54bd978f3 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -901,6 +901,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index ffcc0be7cadc..4940411b8e1c 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1190,9 +1190,11 @@ static inline bool arch_has_pfn_modify_check(void) #ifndef CONFIG_ARCH_HAS_SHSTK #define pte_set_vma_features(pte, vma) pte +#define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9a6b32..eac1ee2f8985 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -608,6 +608,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1250,6 +1251,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1332,6 +1334,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1404,6 +1407,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); From patchwork Thu Jun 6 20:06:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980397 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B9031515 for ; Thu, 6 Jun 2019 20:16:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8AE9728A6D for ; Thu, 6 Jun 2019 20:16:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E60428AA5; Thu, 6 Jun 2019 20:16:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E075D28A6D for ; Thu, 6 Jun 2019 20:16:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D3206B02AB; Thu, 6 Jun 2019 16:15:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 00CD96B02AC; Thu, 6 Jun 2019 16:15:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D07CE6B02AE; Thu, 6 Jun 2019 16:15:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 8D0CF6B02AB for ; Thu, 6 Jun 2019 16:15:32 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id j21so2064390pff.12 for ; Thu, 06 Jun 2019 13:15:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=wRk/MTtMqNJgo/yn0r5hR9R1CE42aqevhtkVjeiHYFY=; b=nEDVgD+UqHKi658Bc6j38ynDf0tHOrRlKu4Z53pjXxxce7XrLqvLkzoGIJnNZij1YB T8pcuhKqzQvjztu8PagqN5AgvF+NuWfYo3xgU3EniwtMso8mlQQqAzz8CrGGR5sTjFNY fLjS6dR2e4ui/vzqrBud+WNorDCeH/rMtzDMBFcVg/bfIIkMREYSqIfjABHgSHnWZclE qmulryEp+5k4fiM+U1/34LbLwEa3RunbcDkqLVAFKKEIZidLjSoHDSeI7iwRd3dFSf7E 2igmVhQl/6BgYD3Xj4WkG1MDOpayqG8+kCiFtKeUfAhAUPw+QV2hXfm8bLG9B2+X9EoO INGw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAUJHVLEXstcmCbv7LZVQHIK16S4Z1NLSmfH1+S5ywxSh/2xqDJ3 J0xYGZqQTMT271TtbzwoCxYvgvzMKUR+spDiyqKhuttV1kBhMXY575Xta0au+k1yQZ5qQ96s048 km6PFoLtSwpqSUOktTyZuZ/Uk0vqTtdUbk9RXskqGmg2ykv6nbm8od8kcS+tnUemOqw== X-Received: by 2002:a63:285:: with SMTP id 127mr293785pgc.200.1559852132165; Thu, 06 Jun 2019 13:15:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqxeB4nXCoYpARrozIB9/JO2hPXYtF0mWzN//FpEH5PyCudnXWpBOgWXo6vD0s5zn/mqLLZl X-Received: by 2002:a63:285:: with SMTP id 127mr293737pgc.200.1559852131302; Thu, 06 Jun 2019 13:15:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852131; cv=none; d=google.com; s=arc-20160816; b=RGivkcJIWKXlOF7uxdny3XaeMx94IpKz92wolzi70rbod6Ezj4pSYj/LYmiPzCLR8U b+vicyb9cgjyjupen3wpGcYjoxl30Iz/6a9SQ1ZCu0i9i4Ko6RCzdVd9N5KxXkyMyM3z nQFjwbrmDMWUlPm+izwGgNaoQxGYIqZgMvEjEH3pIi0ZT+ZEdktEqtI8AJ1oNv/nl2cE W4zuUyDBdzaR2ulPWOTCAppb+Kj7jV3xsZowcYL2iRc3blbSlprSo3ANgQ0d89+ckQTw 5cjeeIN1Vu1YJ2VtLGsjWobk9yd/WgpCpq3raCdm8PYQ6NyAnuj33GhsxHWGZIfYPPC6 0QBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=wRk/MTtMqNJgo/yn0r5hR9R1CE42aqevhtkVjeiHYFY=; b=aWK9kP/hD0sdMTmTaw+lmi3GoUEnLlOvmv0a3F/lcPQXPrzdM1N2tlS0vvq7q7uAbh yWpH7pXJiVs9h3ZbWKl6Vj3FlpwDR+ujYoyT2cS0SEe9mYmwh9waRCeFijncCsK6IW38 m1H0VpU+Kw+OuE09JOdTV5/nlniZfb/rU7aqu1L7atAR2aQUCT3Mib9Ccjn8ML7suVzz DiGuQIcEDyql68EWUBGLJ4k8OS2I5lqSLFf1w+rQzLs4PcN1Cs8khtGUC+VtCG7nig/Y En49+6UixhIBwd2gwioAO2amGMekSDB2aP9v8O4bzUwDmsiFWkwlDS013UMGfgB7uNiY ChEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:31 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:30 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:29 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 17/27] mm: Update can_follow_write_pte/pmd for shadow stack Date: Thu, 6 Jun 2019 13:06:36 -0700 Message-Id: <20190606200646.3951-18-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP can_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW. A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW. Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive. Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?" Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 18 ++++++++++++++++++ include/asm-generic/pgtable.h | 4 ++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 8ff54bd978f3..2a89c168df7b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -913,4 +913,22 @@ inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); } + +inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_dirty_hw(pte); + else + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_dirty_hw(pmd); + else + return pmd_dirty(pmd); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 4940411b8e1c..3324e30bb07f 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1192,10 +1192,14 @@ static inline bool arch_has_pfn_modify_check(void) #define pte_set_vma_features(pte, vma) pte #define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false +#define pte_exclusive(pte, vma) pte_dirty(pte) +#define pmd_exclusive(pmd, vma) pmd_dirty(pmd) #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); +bool pte_exclusive(pte_t pte, struct vm_area_struct *vma); +bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/gup.c b/mm/gup.c index ddde097cf9e4..7d11fff1e8c3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -178,10 +178,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte(pte_t pte, unsigned int flags) +static inline bool can_follow_write(pte_t pte, unsigned int flags, + struct vm_area_struct *vma) { return pte_write(pte) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pte_exclusive(pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma, @@ -219,7 +221,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; - if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) { + if ((flags & FOLL_WRITE) && !can_follow_write(pte, flags, vma)) { pte_unmap_unlock(ptep, ptl); return NULL; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index eac1ee2f8985..d65970b9ece6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1441,10 +1441,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) * FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags) +static inline bool can_follow_write(pmd_t pmd, unsigned int flags, + struct vm_area_struct *vma) { return pmd_write(pmd) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pmd_exclusive(pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, @@ -1457,7 +1459,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, assert_spin_locked(pmd_lockptr(mm, pmd)); - if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags)) + if (flags & FOLL_WRITE && !can_follow_write(*pmd, flags, vma)) goto out; /* Avoid dumping huge zero page */ From patchwork Thu Jun 6 20:06:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980399 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 247516C5 for ; Thu, 6 Jun 2019 20:16:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1291528A6D for ; Thu, 6 Jun 2019 20:16:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 065EA28AA7; Thu, 6 Jun 2019 20:16:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 92BAC28A6D for ; Thu, 6 Jun 2019 20:16:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00EFD6B02AC; Thu, 6 Jun 2019 16:15:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EB2026B02AF; Thu, 6 Jun 2019 16:15:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2CCD6B02B0; Thu, 6 Jun 2019 16:15:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 95EEB6B02AC for ; Thu, 6 Jun 2019 16:15:33 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id z10so2292142pgf.15 for ; Thu, 06 Jun 2019 13:15:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=KLfohGHa7rnp76foE5b4WcKhRrYi/Mci9oJK7pw/J20=; b=Dibui5PF7YbpeK/2XlOP/5ahRPeVMKdDI128EOek0ad3Ili85SSOnGiwquG1InMIXZ YiLQ8TpYYYU5fuDaXdrCIPBevkqT2WQ+hAZMdEAoT+YgwnjWKja3XaezVT/h6m7qyiKd rf7Sc1g0TTEqBLNR3SFAF+jjNtLAnv7T3pyp2uBWtF7K690PUmMe/wq4oeeNYVHbiL4I WKpFPrA1MqlYrYj2BizEApIe3WAu8I934rfbnLVhLjeez9eFmLc2CfdTyYX1AXfCDphg Lpdiw5HpSh6ndORdopgQs1cBdF70BhWIgW/oIZqu5Er5kw82RtmuinNzm4pLw4zxJbBx UeNg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAU9QHTCIboevGLQ2cg8S4F6Aia+VPbyOkSF+tVZgdNH6t+R367h pAHW/dAH+5mG+Az3fzoPvhhhSPWEJu9G7vG84pGcvdah2LCFe9Jo3ud5WzE4vTz/rnwuJA2OIeT N/430AI1B1DA5cEfRpXDsYtgXTeIiT/4HuIQOHBCoj/IT78WocSVOkx1p5LF71zg/ew== X-Received: by 2002:a17:90a:7147:: with SMTP id g7mr1696306pjs.42.1559852133301; Thu, 06 Jun 2019 13:15:33 -0700 (PDT) X-Google-Smtp-Source: APXvYqw2fXFZk+w1oCq6OT4wjnzjrfEySTymv0RAe2pzHrzultVi17cF+Ih68C6hxfdHkpqM8DMv X-Received: by 2002:a17:90a:7147:: with SMTP id g7mr1696247pjs.42.1559852132609; Thu, 06 Jun 2019 13:15:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852132; cv=none; d=google.com; s=arc-20160816; b=k5K8lXMgnyz8cnreMpvRWRIt8y44RCIwlajlT6UkcKnKK3ushcETZ3/HpexJd3pesg JuwAO4kWv0QCZUFPjimtJtAQF7Sd+yoLibziFUeNL2xsoaHyma030oghAAbo+2JdDd2U 2AtIF7CUwhdIeo/HZTp4ngiZz7jFefca5xipopS3ZeN14mLHzNZYHOn+VtcYTwW5SBLI ZeVpAGL8T99LsQsUMlD3QoJfj1ADqoKMOcTWX9pFjwXXpefGmuGvFXyuF8MSHp5DpztN F0c9If1s11dc8tiRBDi9oPVNWJnc33mXX7Y5ekU2OEtR/tzs1cqaFlJ9F4MgjzQtUhns KmwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=KLfohGHa7rnp76foE5b4WcKhRrYi/Mci9oJK7pw/J20=; b=inyfoIGmJfSxYrzD6+5aK7AT+V0uGVV08jHRbTe+TrmiBgNT/5xcOn9pJP62c2fmtP WFVpXtxbxqPb2MVKwQxZHtJntwRk6+uaU7ht4mX2a/HVm3AGipiLLUxK9Oc6LDD8Xnjq GqEwv7JW4rcNgInS9iGrK9PTE8UsJcm3mwKnNzS3vnPPoM4TuSNg/p+EFn5NB++LPqhN Z6/qrTfHHN5ybWreGTRctNYAdy6DxsTDTK7n5JNwfTVJrlCmUIqKbHOnvsACC+MFJ/8f ewmKPvSkEmjaTw9dVpe7fv66qRcHzpIFXhmLITdBmiCV/905/K08YEUUsFUu82KGAvgO GJSw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:32 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:32 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:30 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 18/27] mm: Introduce do_mmap_locked() Date: Thu, 6 Jun 2019 13:06:37 -0700 Message-Id: <20190606200646.3951-19-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 398f1e1c35e5..7cf014604848 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2411,6 +2411,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len) static inline void mm_populate(unsigned long addr, unsigned long len) {} #endif +static inline unsigned long do_mmap_locked(unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags) +{ + struct mm_struct *mm = current->mm; + unsigned long populate; + + down_write(&mm->mmap_sem); + addr = do_mmap(NULL, addr, len, prot, flags, vm_flags, 0, + &populate, NULL); + up_write(&mm->mmap_sem); + + if (populate) + mm_populate(addr, populate); + + return addr; +} + /* These take the mm semaphore themselves */ extern int __must_check vm_brk(unsigned long, unsigned long); extern int __must_check vm_brk_flags(unsigned long, unsigned long, unsigned long); From patchwork Thu Jun 6 20:06:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980401 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C55B1515 for ; Thu, 6 Jun 2019 20:16:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1907A28A6D for ; Thu, 6 Jun 2019 20:16:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0CB7C28AA7; Thu, 6 Jun 2019 20:16:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C51A28A6D for ; Thu, 6 Jun 2019 20:16:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 627786B02AF; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5DA176B02B1; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47C096B02B2; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 037716B02AF for ; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id g38so2275826pgl.22 for ; Thu, 06 Jun 2019 13:15:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=clcsm5axWe8+hZG2s5xT0CCWLgJnsuYEYwCwmIqgIZ0=; b=iVkMzjN06wN4a0t6zh3StQqRDdGgRa8R40OroeMNFUkSbocwSAjTw69tpkqlZRhgK6 EePnmmwTBMvF+mwbSdUSsOZCT1RK4fH8Oh+RmGb2sD92eP2tBKC2do+ygcjmXnHeB5OC d7EnbdKO7S0yes8GsE3492i4KY4GLRDm3uIHEWakBr+05TQ5zHgVveF4hdWcYgJ4efl4 DlDj/x94PK8fM4CI4hMNsYaGdBY8gRR8umvI6A5ogra1m8UpnAHL1xy6d9f3LGpKUNhs DMy0le9xnG8QYX2FROgDQBOIxKKGxDqAg82D4uFOQCJcKXt5Mr2zjlIS5J/C+cqq9r1o 4dAw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAW4nCC8WIxRs6o2VES1qkUWnyjwNFy4s8E4lqVYpf/LTgURZ2EW TKnzXfKlnXn57AlJ/R4A3MwyOr8YyvVM4zFHNbF8eXF8dVPl18MtYzZl+L8z+CUEvccaZW3szO9 3z5pi77UnTtiWmui90aE8xCErClvhIHbvtK7o5YOqGVzSIHSCYopkyeGzwzPdU8HRFA== X-Received: by 2002:a17:902:b70b:: with SMTP id d11mr53038053pls.84.1559852135631; Thu, 06 Jun 2019 13:15:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqwz6QlsgGHnxqsrsnwryWEjL8wdaRq2HlgkXY59vtBfZCwqO6YNrNnkQWUmfLG/gjYUZgDE X-Received: by 2002:a17:902:b70b:: with SMTP id d11mr53037948pls.84.1559852133959; Thu, 06 Jun 2019 13:15:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852133; cv=none; d=google.com; s=arc-20160816; b=C8CvqoJWjfrRW4btg2jyYnpddLlM5Iu3GvXxslqhNIvWi/NVUv4ze6xN1NPsSndXWo fjmKZINuh2aBspBHqT8cxiWT8AR/PI0NTsSdsAiamwUgwNdAaXSFYtx4AA6pVpy1vCOF SM9CC7znJAXeQx6qx4L91XuDuL1ku8FONKKx75z11JilUpb9iinMeBKCqrx9+Am1YXNJ uhU0cdtjFOG0reqCG1Ex5uyflqjTaGDVg6hF+8xmOs6Mr21S6IBO/vSOUdoRsAaFNcLd mMF+k+dUzT6BLVEkBV6Fz/+HQLzlZsGPz7wp2Z4gsdwbTUDGO9hdRcusxm1lIB7wnIrG 85+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=clcsm5axWe8+hZG2s5xT0CCWLgJnsuYEYwCwmIqgIZ0=; b=USLyz5LGBwRu41rKiJ7l1uFhSar/hmp6cnrUU6bHwWPCm/nBHHVzlcKUvux34LEWd3 Yvi4gKGcDZ9cixq1MstjzEuRJJ1wPUEe1zPEGZDfoxLaFs0/6vel7fgot+ApdzaKVYGZ rSduwrAwrZwgOm0/wHtadh25bggrsYobz+BGKeG5HKu7JE86gBEq4hMkZom74QKXr1QV OowC0lPDCuxvOsop2rornUDMIptQHwnnu2ThVN762vEtpU7nfsbFk2C+YttRtSKCcoRP Br9dM5ABogQX70jYZaUaCnk/0Kl8NXDzfbs+HWuqrHrIFF06Q636B6MtmMNqcW9FJZfV HskQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:33 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:33 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:32 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 19/27] x86/cet/shstk: User-mode shadow stack support Date: Thu, 6 Jun 2019 13:06:38 -0700 Message-Id: <20190606200646.3951-20-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. It has a fixed size of RLIMIT_STACK. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 34 +++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 18 +++ arch/x86/include/asm/processor.h | 5 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 116 ++++++++++++++++++ arch/x86/kernel/cpu/common.c | 25 ++++ arch/x86/kernel/process.c | 1 + .../arch/x86/include/asm/disabled-features.h | 8 +- 9 files changed, 215 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h new file mode 100644 index 000000000000..c952a2ec65fe --- /dev/null +++ b/arch/x86/include/asm/cet.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; +/* + * Per-thread CET status + */ +struct cet_status { + unsigned long shstk_base; + unsigned long shstk_size; + unsigned int shstk_enabled:1; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk(void); +void cet_disable_shstk(void); +void cet_disable_free_shstk(struct task_struct *p); +#else +static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline void cet_disable_shstk(void) {} +static inline void cet_disable_free_shstk(struct task_struct *p) {} +#endif + +#define cpu_x86_cet_enabled() \ + (cpu_feature_enabled(X86_FEATURE_SHSTK) || \ + cpu_feature_enabled(X86_FEATURE_IBT)) + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index a5ea841cc6d2..06323ebed643 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -62,6 +62,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -81,7 +87,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 979ef971cc78..30e9107974fa 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -839,4 +839,22 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0 /* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2 /* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4 /* kernel shstk pointer */ +#define MSR_IA32_PL1_SSP 0x6a5 /* ring-1 shstk pointer */ +#define MSR_IA32_PL2_SSP 0x6a6 /* ring-2 shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7 /* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8 /* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL +#define MSR_IA32_CET_WAIT_ENDBR 0x00000000000000800UL +#define MSR_IA32_CET_BITMAP_MASK 0xfffffffffffff000ULL + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index c34a35c78618..2ae7c1bf4e43 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -24,6 +24,7 @@ struct vm86; #include #include #include +#include #include #include @@ -487,6 +488,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET + struct cet_status cet; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index ce1b5cc360a2..584ed7e9a599 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -140,6 +140,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..2a48634aa6ce --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,116 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * cet.c - Control-flow Enforcement (CET) + * + * Copyright (c) 2018, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int set_shstk_ptr(unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + modify_fpu_regs_end(); + return 0; +} + +static unsigned long get_shstk_addr(void) +{ + unsigned long ptr; + + if (!current->thread.cet.shstk_enabled) + return 0; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_PL3_SSP, ptr); + modify_fpu_regs_end(); + return ptr; +} + +int cet_setup_shstk(void) +{ + unsigned long addr, size; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EOPNOTSUPP; + + size = rlimit(RLIMIT_STACK); + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + /* + * Return actual error from do_mmap(). + */ + if (addr >= TASK_SIZE_MAX) + return addr; + + set_shstk_ptr(addr + size - sizeof(u64)); + current->thread.cet.shstk_base = addr; + current->thread.cet.shstk_size = size; + current->thread.cet.shstk_enabled = 1; + return 0; +} + +void cet_disable_shstk(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_SHSTK_EN); + wrmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, 0); + modify_fpu_regs_end(); + current->thread.cet.shstk_enabled = 0; +} + +void cet_disable_free_shstk(struct task_struct *tsk) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || + !tsk->thread.cet.shstk_enabled) + return; + + if (tsk->mm && (tsk == current)) + cet_disable_shstk(); + + /* + * Free only when tsk is current or shares mm + * with current but has its own shstk. + */ + if (tsk->mm && (tsk->mm == current->mm) && + (tsk->thread.cet.shstk_base)) { + vm_munmap(tsk->thread.cet.shstk_base, + tsk->thread.cet.shstk_size); + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + } + + tsk->thread.cet.shstk_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 2c57fffebf9b..b0780fe8717e 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -53,6 +53,7 @@ #include #include #include +#include #ifdef CONFIG_X86_LOCAL_APIC #include @@ -417,6 +418,29 @@ static __init int setup_disable_pku(char *arg) __setup("nopku", setup_disable_pku); #endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ + if (cpu_x86_cet_enabled()) + cr4_set_bits(X86_CR4_CET); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk(char *s) +{ + /* require an exact match without trailing characters */ + if (s[0] != '\0') + return 0; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_SHSTK); + pr_info("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n"); + return 1; +} +__setup("no_cet_shstk", setup_disable_shstk); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization @@ -1393,6 +1417,7 @@ static void identify_cpu(struct cpuinfo_x86 *c) x86_init_rdrand(c); x86_init_cache_qos(c); setup_pku(c); + setup_cet(c); /* * Clear/Set all flags overridden by options, need do it diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index d360bf4d696b..a4deb79b1089 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -42,6 +42,7 @@ #include #include #include +#include #include "process.h" diff --git a/tools/arch/x86/include/asm/disabled-features.h b/tools/arch/x86/include/asm/disabled-features.h index a5ea841cc6d2..06323ebed643 100644 --- a/tools/arch/x86/include/asm/disabled-features.h +++ b/tools/arch/x86/include/asm/disabled-features.h @@ -62,6 +62,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -81,7 +87,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) From patchwork Thu Jun 6 20:06:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980403 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33C8A1515 for ; Thu, 6 Jun 2019 20:16:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2279028A6D for ; Thu, 6 Jun 2019 20:16:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 167BC28AA7; Thu, 6 Jun 2019 20:16:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F05828A6D for ; Thu, 6 Jun 2019 20:16:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D54836B02B1; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CDDF26B02B2; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B56036B02B4; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 7824F6B02B2 for ; Thu, 6 Jun 2019 16:15:36 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id b24so2141696plz.20 for ; Thu, 06 Jun 2019 13:15:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uWb7t+eKjZRdrxDegnwMc3Dgz6fTxwnmp/IT3TakFdo=; b=dymfRaboZnTiHaW7YUEYkUteDOOqmL1ejWaE8N631ixXy3hXiiJqqjjnQwsU4CT2KT 78t3iXf5bZCmjnPp46aOUzn1YLQtxPfvINU1xKbgEhhx5Lkdc1Eip+0WtvM9Nw0vRGZp 94kaTzA2EhXr6jIq6GcZcnrfBZKTvxQ1P12cGNJy6/W8SZMOYNITCNvRtyyEJTDD9neD LtDfgWw2gxXMoyI1YMSZiuq/Y8yuW/NAuMRj0gXJ/nBEjTi+8wHSrkKz/QnPfKKRNK3T WrAPmHypJZZyU0MDfs73LItougslg7RAvp8nwL3LTzRQrqg6Zse0uLeJ7gDZ+GDUCr8x m8JA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWDlWI3bja8EvfC56toMrNZ3x4vsTdD25fYQ7tEhDT0sdq+WXJU lgT+jfWAzsxMZgtwolfO+/ZkEWLuFTAVO6zvknfrfNgjHrkOAKnTHvvRWQMm1RFb0nZHYJNcn8I Cv+OhSoscEKDAGMDUAKWoqIavYVIfRVzeWgS3zSozZ7K/1bTgpC/iTKPtnpNzh4xOmA== X-Received: by 2002:a17:90a:d582:: with SMTP id v2mr1630352pju.22.1559852136133; Thu, 06 Jun 2019 13:15:36 -0700 (PDT) X-Google-Smtp-Source: APXvYqxxsaXfsbiQEEwhpiYuqSKRwJ5iCbWgbR0A5TIc7QpIDPJig33wZOrbDSIsk1KTC+NQz1M9 X-Received: by 2002:a17:90a:d582:: with SMTP id v2mr1630300pju.22.1559852135227; Thu, 06 Jun 2019 13:15:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852135; cv=none; d=google.com; s=arc-20160816; b=ExjYwi01vm+WGDLWZNriht8zB0Jw7EGbbeYivWpG1PA4MWjSKNhfSGUmV5H/W9FNBX Vzhi36pmeAN5AUe96RsYPHInQPytG19ywyScH16nre2N8GgbqJZGfaHMDzha3Mq7V44R iSw8G37EklTxaC1Y04HDt/vRUwnkRj+ZvIhJ5TbyAD/AAJFiCDWfmWqt/JCkzwVyhvEk 5cZkSS7O2MPHT1nex6lSxaEMWxyZpxZGZlkQY2N7GWwOnXLMnpR72sONOuvRNyMenHdR T4z4FediQYynsyND8Vgy+VU/8QbX9IqnIike35BiSUGPgEjDecgxVucJ9TRnQcX0JP0k BZ7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uWb7t+eKjZRdrxDegnwMc3Dgz6fTxwnmp/IT3TakFdo=; b=j0hnoQhwsRwarfTdOQKpVvNu5hvUzlTRtbUQDh4AYdxBYtTnk35oFXxXWZT70plz8d 34KDfRJDO1SFeRyg9M4g/lVNdD/lgVrjRyTXIU9gza4ZW4/V3T7fvfMWBGxJQLja9gdf zEuN+t3dKFcE1sCk3l4OGQ2N47IY4BcKSVKs6IIZzLGN07RNNMOFn88Oe4yJc5mhP73Y NlRMRBsGZ553DKs9BxcZUo+jnwaqG4hb5nwr+4UfccyeO7RYIRdy0Sphaly2HJsRqDq6 hnEcxBvV6b0knvdx79SXbZtalo1D359EfFk5J3Vr7pCjB/21ZS+9xfYPUqdMCaeMTPit ZMfQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:35 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:34 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:33 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 20/27] x86/cet/shstk: Introduce WRUSS instruction Date: Thu, 6 Jun 2019 13:06:39 -0700 Message-Id: <20190606200646.3951-21-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler. This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does a fixup. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 0a3c4cab39db..99530ab0a077 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -250,6 +250,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + asm_volatile_goto("1: wrussd %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + WARN_ONCE(1, "%s used but not supported.\n", __func__); + return -EFAULT; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{ + asm_volatile_goto("1: wrussq %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#endif /* CONFIG_X86_INTEL_CET */ + #define nop() asm volatile ("nop") From patchwork Thu Jun 6 20:06:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980405 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5EA916C5 for ; Thu, 6 Jun 2019 20:16:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CDCA28A6D for ; Thu, 6 Jun 2019 20:16:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4009C28AA8; Thu, 6 Jun 2019 20:16:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B09428A6D for ; Thu, 6 Jun 2019 20:16:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3914D6B02B4; Thu, 6 Jun 2019 16:15:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 31BDA6B02B5; Thu, 6 Jun 2019 16:15:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F6356B02B6; Thu, 6 Jun 2019 16:15:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id B72136B02B4 for ; Thu, 6 Jun 2019 16:15:38 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id x9so2594349pfm.16 for ; Thu, 06 Jun 2019 13:15:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=yi3r3LHXHUPKxuht62yABB7GgHx0wdi7t5SAXLoiMTc=; b=NBmW7qDCtulEAnHGls/qX/txSSHRjU3aAQ0MbEleZjSs4SC3dfKbApX0pl9vENOp3N 8Zb0P3fXe4jRMIgDpNUR2abVeSes1nZEgHZquC5RsiFrLKY1hLxe/U0PvaRs5QIpTzd8 6RHlPNVwE1GKYjqwSBw5//aa4RSreg3fknh/w7CziGYegHldzuNBRPUb3ivV610MWjKC OMbcR3peor+ipCC0iKOBIj9LW5U+Asn2i75XaYCfYYNdsS69WpEIjoCdGncpDq2ssIhw g/wyyjHLDiyRi5UA67VlFicewb45N+RDHkzQd6CjDjWRJ3tXcqAf1j3ccClUJVe8qh36 U3vQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVqczkdCXRahRNjUO0fyIvLWjztm8Vu51Mn7pQ15xU4NRFzXLKw G5QdZCryUXqNZ+jR0JPjyO5H0tjBH6v0uCRV5ygQ9oa9Heju9gjx/5WIElMK8EHNM92ZzZv0MSn vneAHgF889zzWqKanfpIDj9Q5/hNUwqFrEfDiu7Gvs2c4hfMPBdB7Vq4E+S6I3jY3/Q== X-Received: by 2002:a17:902:2983:: with SMTP id h3mr8055982plb.45.1559852138373; Thu, 06 Jun 2019 13:15:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqznwGoijd1TuevW1gSp+PIA+ffMGjWwrfog5FOMUvW68ngwnIo9Geo/x7vNOnNBtHdbjbl3 X-Received: by 2002:a17:902:2983:: with SMTP id h3mr8055866plb.45.1559852136693; Thu, 06 Jun 2019 13:15:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852136; cv=none; d=google.com; s=arc-20160816; b=V4mV3bxhdss4GsLYd44IBV+2ZlfaNbm9+g3LMpMwLdjv96SlzAwOZWl+pYBkvnmDRw 9GW9L818nfd4UY0N6r78n6upNizmh4ZD0CC07aevQizhOJC6keqFPrC/blIMcjy6M4eP v5KUhrXZsXsvk++IP/YJlsrEC/20ZowCDRhSwqWFxUBHpVMue5Y+5J7sJngg9N/EkP/9 +KnJ7YDFpUedrTZEUu2BYC8IOJTq0bLNYoNs2bDpHsHfqRpWGTD9b0DWbQ8hJrUMNdZw yC1lllpJbVCCuz4C6OaXnarRBhDFvHwmk0P+0GJGMBkIKeAP7XpTvnCJ3Fxa1+w1NHMA i0tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=yi3r3LHXHUPKxuht62yABB7GgHx0wdi7t5SAXLoiMTc=; b=yiQ3INbVj6jG5t5X6S+6Bp2E4jkYHPaSKEudEVTMDnMoY7KKx76s4Hy7+0a9OIDkOo BUBr00lqBKgWGdev+uHtL5Uhgd5WG1QsD7LpHWZAwcCoE+EdC2gXz+SHbQSwU0B0Dd2e TK3lNV6JKPsmBTUZniURRXPm4DbiPLEJs58sbFCb+1ZDRCsRqXWU63gVQrYHGxCSLg/j k+/9YqRnW3MjJREsGsDu49NQ0fpCJpzWdWMhBjZpPMfQU7pKJlVqi1Y+wlkj9WYhoTss QtsYNGphKX/eRtAVNfeciMQlZ5BAVfKqTuznX5GhRf2QhzP1KBT8rAkBJ/V45vP1h10+ enmw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:36 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:36 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:34 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 21/27] x86/cet/shstk: Handle signals for shadow stack Date: Thu, 6 Jun 2019 13:06:40 -0700 Message-Id: <20190606200646.3951-22-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer. Signed-off-by: Yu-cheng Yu --- arch/x86/ia32/ia32_signal.c | 8 ++ arch/x86/include/asm/cet.h | 7 ++ arch/x86/include/asm/fpu/internal.h | 2 + arch/x86/include/asm/fpu/signal.h | 2 + arch/x86/include/uapi/asm/sigcontext.h | 15 +++ arch/x86/kernel/cet.c | 141 +++++++++++++++++++++++++ arch/x86/kernel/fpu/signal.c | 67 ++++++++++++ arch/x86/kernel/signal.c | 8 ++ 8 files changed, 250 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 629d1ee05599..5216ca782262 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -34,6 +34,7 @@ #include #include #include +#include /* * Do a signal return; undo the signal stack. @@ -235,6 +236,7 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, ksig->ka.sa.sa_restorer) sp = (unsigned long) ksig->ka.sa.sa_restorer; + sp = fpu__alloc_sigcontext_ext(sp); sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, @@ -295,6 +297,9 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, restorer = &frame->retcode; } + if (setup_fpu_system_states(1, (unsigned long)restorer, fpstate)) + return -EFAULT; + put_user_try { put_user_ex(ptr_to_compat(restorer), &frame->pretcode); @@ -384,6 +389,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_fpu_system_states(1, (unsigned long)restorer, fpstate); + if (err) return -EFAULT; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index c952a2ec65fe..422ccb8adbb7 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -6,6 +6,8 @@ #include struct task_struct; +struct sc_ext; + /* * Per-thread CET status */ @@ -19,10 +21,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); +int cet_restore_signal(bool ia32, struct sc_ext *sc); +int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); #else static inline int cet_setup_shstk(void) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} +static inline int cet_restore_signal(bool ia32, struct sc_ext *sc) { return -EINVAL; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + struct sc_ext *sc) { return -EINVAL; } #endif #define cpu_x86_cet_enabled() \ diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 148a3d8c8c35..7db8c19c9072 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -472,6 +472,8 @@ static inline void copy_kernel_to_fpregs(union fpregs_state *fpstate) __copy_kernel_to_fpregs(fpstate, -1); } +extern int setup_fpu_system_states(int is_ia32, unsigned long restorer, + void __user *fp); extern int copy_fpstate_to_sigframe(void __user *buf, void __user *fp, int size); /* diff --git a/arch/x86/include/asm/fpu/signal.h b/arch/x86/include/asm/fpu/signal.h index 7fb516b6893a..630a658aeea3 100644 --- a/arch/x86/include/asm/fpu/signal.h +++ b/arch/x86/include/asm/fpu/signal.h @@ -25,6 +25,8 @@ extern void convert_from_fxsr(struct user_i387_ia32_struct *env, extern void convert_to_fxsr(struct fxregs_state *fxsave, const struct user_i387_ia32_struct *env); +unsigned long fpu__alloc_sigcontext_ext(unsigned long sp); + unsigned long fpu__alloc_mathframe(unsigned long sp, int ia32_frame, unsigned long *buf_fx, unsigned long *size); diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/uapi/asm/sigcontext.h index 844d60eb1882..e3b08d1c0d3b 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,21 @@ struct _xstate { /* New processor state extensions go here: */ }; +/* + * Sigcontext extension (struct sc_ext) is located after + * sigcontext->fpstate. Because currently only the shadow + * stack pointer is saved there and the shadow stack depends + * on XSAVES, we can find sc_ext from sigcontext->fpstate. + * + * The 64-bit fpstate has a size of fpu_user_xstate_size, plus + * FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext + * is located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; +}; + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 2a48634aa6ce..b247cd15c1e2 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -19,6 +19,8 @@ #include #include #include +#include +#include static int set_shstk_ptr(unsigned long addr) { @@ -51,6 +53,80 @@ static unsigned long get_shstk_addr(void) return ptr; } +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64(token) ((token & TOKEN_MODE_MASK) == TOKEN_MODE_64) +#define IS_TOKEN_32(token) ((token & TOKEN_MODE_MASK) == 0) + +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +static int verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp = 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (!ia32 && !IS_TOKEN_64(token)) + return -EINVAL; + else if (ia32 && !IS_TOKEN_32(token)) + return -EINVAL; + + token &= ~TOKEN_MODE_MASK; + + /* + * Restore address properly aligned? + */ + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + /* + * Token was placed properly? + */ + if ((ALIGN_DOWN(token, 8) - 8) != ssp) + return -EINVAL; + + *new_ssp = token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp = 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr = ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |= TOKEN_MODE_64; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -114,3 +190,68 @@ void cet_disable_free_shstk(struct task_struct *tsk) tsk->thread.cet.shstk_enabled = 0; } + +/* + * Called from __fpu__restore_sig() under the protection + * of fpregs_lock(). + */ +int cet_restore_signal(bool ia32, struct sc_ext *sc_ext) +{ + unsigned long new_ssp = 0; + u64 msr_ia32_u_cet = 0; + int err; + + if (current->thread.cet.shstk_enabled) { + err = verify_rstor_token(ia32, sc_ext->ssp, &new_ssp); + if (err) + return err; + + msr_ia32_u_cet |= MSR_IA32_CET_SHSTK_EN; + } + + wrmsrl(MSR_IA32_PL3_SSP, new_ssp); + wrmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); + return 0; +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, struct sc_ext *sc_ext) +{ + unsigned long ssp = 0, new_ssp = 0; + u64 msr_ia32_u_cet = 0; + int err; + + msr_ia32_u_cet = 0; + ssp = 0; + + if (current->thread.cet.shstk_enabled) { + ssp = get_shstk_addr(); + err = create_rstor_token(ia32, ssp, &new_ssp); + if (err) + return err; + + if (ia32) { + ssp = new_ssp - sizeof(u32); + err = write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp = new_ssp - sizeof(u64); + err = write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + msr_ia32_u_cet |= MSR_IA32_CET_SHSTK_EN; + sc_ext->ssp = new_ssp; + } + + modify_fpu_regs_begin(); + wrmsrl(MSR_IA32_PL3_SSP, ssp); + wrmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); + modify_fpu_regs_end(); + return 0; +} diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index e38b272793b1..0a67518c142f 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -51,6 +51,58 @@ static inline int check_for_xstate(struct fxregs_state __user *buf, return 0; } +int setup_fpu_system_states(int is_ia32, unsigned long restorer, + void __user *fp) +{ + int err = 0; + +#ifdef CONFIG_X86_64 + if (cpu_x86_cet_enabled() && fp) { + struct sc_ext ext = {0, 0}; + + err = cet_setup_signal(is_ia32, restorer, &ext); + if (!err) { + void __user *p; + + ext.total_size = sizeof(ext); + + p = fp + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_to_user(p, &ext, sizeof(ext))) + return -EFAULT; + } + } +#endif + + return err; +} + +static int restore_fpu_system_states(int is_ia32, void __user *fp) +{ + int err = 0; + +#ifdef CONFIG_X86_64 + if (cpu_x86_cet_enabled() && fp) { + struct sc_ext ext = {0, 0}; + void __user *p; + + p = fp + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_from_user(&ext, p, sizeof(ext))) + return -EFAULT; + + if (ext.total_size != sizeof(ext)) + return -EFAULT; + + err = cet_restore_signal(is_ia32, &ext); + } +#endif + + return err; +} + /* * Signal frame handlers. */ @@ -349,6 +401,10 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) pagefault_disable(); ret = copy_user_to_fpregs_zeroing(buf_fx, xfeatures, fx_only); pagefault_enable(); + + if (!ret) + ret = restore_fpu_system_states(0, buf); + if (!ret) { fpregs_mark_activate(); fpregs_unlock(); @@ -435,6 +491,17 @@ int fpu__restore_sig(void __user *buf, int ia32_frame) return __fpu__restore_sig(buf, buf_fx, size); } +unsigned long fpu__alloc_sigcontext_ext(unsigned long sp) +{ + /* + * sigcontext_ext is at: fpu + fpu_user_xstate_size + + * FP_XSTATE_MAGIC2_SIZE, then aligned to 8. + */ + if (cpu_x86_cet_enabled()) + sp -= (sizeof(struct sc_ext) + 8); + return sp; +} + unsigned long fpu__alloc_mathframe(unsigned long sp, int ia32_frame, unsigned long *buf_fx, unsigned long *size) diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 3b0dcec597ce..c81192bdc96c 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include +#include #define COPY(x) do { \ get_user_ex(regs->x, &sc->x); \ @@ -264,6 +265,7 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, sp = (unsigned long) ka->sa.sa_restorer; } + sp = fpu__alloc_sigcontext_ext(sp); sp = fpu__alloc_mathframe(sp, IS_ENABLED(CONFIG_X86_32), &buf_fx, &math_size); *fpstate = (void __user *)sp; @@ -493,6 +495,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig, err |= setup_sigcontext(&frame->uc.uc_mcontext, fp, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_fpu_system_states(0, (unsigned long)ksig->ka.sa.sa_restorer, fp); + if (err) return -EFAULT; @@ -579,6 +584,9 @@ static int x32_setup_rt_frame(struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_fpu_system_states(0, (unsigned long)ksig->ka.sa.sa_restorer, fpstate); + if (err) return -EFAULT; From patchwork Thu Jun 6 20:06:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980407 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 266231515 for ; Thu, 6 Jun 2019 20:16:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1502428A6D for ; Thu, 6 Jun 2019 20:16:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 07F5328AA8; Thu, 6 Jun 2019 20:16:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF94628A6D for ; Thu, 6 Jun 2019 20:16:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 256AB6B02B5; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1E05F6B02B7; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F26D76B02B8; Thu, 6 Jun 2019 16:15:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id B15956B02B5 for ; Thu, 6 Jun 2019 16:15:39 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id x9so2594381pfm.16 for ; Thu, 06 Jun 2019 13:15:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=T05HjB1dOtC+7biOT8TU+T3c2hChEvbbYH9fK1G4wmY=; b=uhUWkTXGpncXz+z9i7XWp4xKmd9+1+xGWazM4VdbyKG1fUU5W6nwXc2pE9gNKA4t9W m4WZazk05XGgyXMVz8Sou9rz59Zvh6A80LcQuNYZ4lciwFTUSSuCQIFBaFuSk+GcEU9j OvSVr6mGS5WJaALROzKRN1dglpceMiUiFEQC78N0FGnABNxbQIOAisRYYCjauB87uBH2 p3A4yPUANH4srp0nHLJUxRJEpDoqBrL33li0NifZjkjw/r6U2x/XEbOoSkulQQIV/KRn bzSAkwY1ziN8cyDP5bLx60shBdc14tr4fW8Nj8Z9HqtslH1QFEGUASZQtyv70/CowR4g iyIw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWVwBcNnLs1/eauhjTCX3DZV9J6W2LgZ649RQ5C2iP0cvx8cBzo MzsBuu5kUhS+/FKmLXJ+jAmFE+SQYDu/6CalOzfVLlSF21u0cbeyapVJIpSYgA7e66EUoTDkbH/ GzZCHR03PYw+r3L7ZzI7+lpUHhXXiM15vNlyUzH5OVYCvtBGfzZCAjdLHmVkUlf7b3g== X-Received: by 2002:a63:3d89:: with SMTP id k131mr299710pga.121.1559852139247; Thu, 06 Jun 2019 13:15:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqxnyS2ITahLPu0ZqvFbqFSftDnHjRkzkQGczhakqfzswXIdcDkKfoh7VMyLfC4WHYklMzpU X-Received: by 2002:a63:3d89:: with SMTP id k131mr299635pga.121.1559852137964; Thu, 06 Jun 2019 13:15:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852137; cv=none; d=google.com; s=arc-20160816; b=zUWx5sKKlX6ULotYRCZE2tqG2CyoFrihk/6nsDjZSoc8OJdfcwA1BYqrZI3ALaqp4Q zjNGOh8HByMYuYyO6XYmekOjkotjzt24hAbCEwBKk56WncnQr1eJm+UjjVtfWsOwNS+T RQoKtGLgLxBoRXpBv6IDc7sMJ9Dc1ELk1HXd17793mI/B25/RzFbm/YDL7GP7RxWDhji YSMewIHD67B6cUNsMHSXULy/Mnnn9kziHN00bI/lMWrmo/4Mm4FIcG6xOiF9ZAUU3HCP gGXCry+mwdBctv1er377wJx1SQ+ISR1I8qSnpNWFud5U5fIqAWqfA+jtfDZPnUWVDBQX bnAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=T05HjB1dOtC+7biOT8TU+T3c2hChEvbbYH9fK1G4wmY=; b=Xkj2UV3GyDCzhoVcBIUlN3llh10kdAqQPYHwqEZhAVMGATgopTCLLYmIJ/5M0YE+c6 RYZ9l8IL7oNtD7xJ1LEoaKdpyfX4Sw2Ff7iOowovYOHulAmLS3z60qvIlKBAKSeg40PO lAkx34REd0e97MOdspDgKzkvfSCiY7voCsNQpdkqgsQftrmz+U+stCchdQneuHEPV3AY EYEL7gzHKToU0cotujdyTQSlloUD0sLcJ2oH21dW8FVkpuQwx05SvkZW5aITks46ezly exh/Ygj4q7N9x9plPAc2wJzpcY4gKqzNjEFEW/0H/RbAkqmjSuJSXbpOA41wPUxPf8X+ gxOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:37 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:37 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:36 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 22/27] binfmt_elf: Extract .note.gnu.property from an ELF file Date: Thu, 6 Jun 2019 13:06:41 -0700 Message-Id: <20190606200646.3951-23-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP An ELF file's .note.gnu.property indicates features the executable file can support. For example, the property GNU_PROPERTY_X86_FEATURE_1_AND indicates the file supports GNU_PROPERTY_X86_FEATURE_1_IBT and/or GNU_PROPERTY_X86_FEATURE_1_SHSTK. With this patch, if an arch needs to setup features from ELF properties, it needs CONFIG_ARCH_USE_GNU_PROPERTY to be set, and a specific arch_setup_property(). For example, for X86_64: int arch_setup_property(void *ehdr, void *phdr, struct file *f, bool inter) { int r; uint32_t property; r = get_gnu_property(ehdr, phdr, f, GNU_PROPERTY_X86_FEATURE_1_AND, &property); ... } Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- fs/Kconfig.binfmt | 3 + fs/Makefile | 1 + fs/binfmt_elf.c | 13 ++ fs/gnu_property.c | 351 +++++++++++++++++++++++++++++++++++++++ include/linux/elf.h | 12 ++ include/uapi/linux/elf.h | 14 ++ 6 files changed, 394 insertions(+) create mode 100644 fs/gnu_property.c diff --git a/fs/Kconfig.binfmt b/fs/Kconfig.binfmt index f87ddd1b6d72..397138ab305b 100644 --- a/fs/Kconfig.binfmt +++ b/fs/Kconfig.binfmt @@ -36,6 +36,9 @@ config COMPAT_BINFMT_ELF config ARCH_BINFMT_ELF_STATE bool +config ARCH_USE_GNU_PROPERTY + bool + config BINFMT_ELF_FDPIC bool "Kernel support for FDPIC ELF binaries" default y if !BINFMT_ELF diff --git a/fs/Makefile b/fs/Makefile index c9aea23aba56..b69f18c14e09 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -44,6 +44,7 @@ obj-$(CONFIG_BINFMT_ELF) += binfmt_elf.o obj-$(CONFIG_COMPAT_BINFMT_ELF) += compat_binfmt_elf.o obj-$(CONFIG_BINFMT_ELF_FDPIC) += binfmt_elf_fdpic.o obj-$(CONFIG_BINFMT_FLAT) += binfmt_flat.o +obj-$(CONFIG_ARCH_USE_GNU_PROPERTY) += gnu_property.o obj-$(CONFIG_FS_MBCACHE) += mbcache.o obj-$(CONFIG_FS_POSIX_ACL) += posix_acl.o diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 8264b468f283..c3ea73787e93 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1080,6 +1080,19 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } + if (interpreter) { + retval = arch_setup_property(&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + } else { + retval = arch_setup_property(&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + + if (retval < 0) + goto out_free_dentry; + if (interpreter) { unsigned long interp_map_addr = 0; diff --git a/fs/gnu_property.c b/fs/gnu_property.c new file mode 100644 index 000000000000..9c4d1d5ebf00 --- /dev/null +++ b/fs/gnu_property.c @@ -0,0 +1,351 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Extract an ELF file's .note.gnu.property. + * + * The path from the ELF header to the note section is the following: + * elfhdr->elf_phdr->elf_note->property[]. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The .note.gnu.property layout: + * + * struct elf_note { + * u32 n_namesz; --> sizeof(n_name[]); always (4) + * u32 n_ndescsz;--> sizeof(property[]) + * u32 n_type; --> always NT_GNU_PROPERTY_TYPE_0 + * }; + * char n_name[4]; --> always 'GNU\0' + * + * struct { + * struct gnu_property { + * u32 pr_type; + * u32 pr_datasz; + * }; + * u8 pr_data[pr_datasz]; + * }[]; + */ + +#define BUF_SIZE (PAGE_SIZE / 4) + +typedef bool (test_item_fn)(void *buf, u32 *arg, u32 type); +typedef void *(next_item_fn)(void *buf, u32 *arg, u32 type); + +static inline bool test_note_type(void *buf, u32 *align, u32 note_type) +{ + struct elf_note *n = buf; + + return ((n->n_type == note_type) && (n->n_namesz == 4) && + (memcmp(n + 1, "GNU", 4) == 0)); +} + +static inline void *next_note(void *buf, u32 *align, u32 note_type) +{ + struct elf_note *n = buf; + u64 size; + + if (check_add_overflow((u64)sizeof(*n), (u64)n->n_namesz, &size)) + return NULL; + + size = round_up(size, *align); + + if (check_add_overflow(size, (u64)n->n_descsz, &size)) + return NULL; + + size = round_up(size, *align); + + if (buf + size < buf) + return NULL; + else + return (buf + size); +} + +static inline bool test_property(void *buf, u32 *max_type, u32 pr_type) +{ + struct gnu_property *pr = buf; + + /* + * Property types must be in ascending order. + * Keep track of the max when testing each. + */ + if (pr->pr_type > *max_type) + *max_type = pr->pr_type; + + return (pr->pr_type == pr_type); +} + +static inline void *next_property(void *buf, u32 *max_type, u32 pr_type) +{ + struct gnu_property *pr = buf; + + if ((buf + sizeof(*pr) + pr->pr_datasz < buf) || + (pr->pr_type > pr_type) || + (pr->pr_type > *max_type)) + return NULL; + else + return (buf + sizeof(*pr) + pr->pr_datasz); +} + +/* + * Scan 'buf' for a pattern; return true if found. + * *pos is the distance from the beginning of buf to where + * the searched item or the next item is located. + */ +static int scan(u8 *buf, u32 buf_size, int item_size, test_item_fn test_item, + next_item_fn next_item, u32 *arg, u32 type, u32 *pos) +{ + int found = 0; + u8 *p, *max; + + max = buf + buf_size; + if (max < buf) + return 0; + + p = buf; + + while ((p + item_size < max) && (p + item_size > buf)) { + if (test_item(p, arg, type)) { + found = 1; + break; + } + + p = next_item(p, arg, type); + } + + *pos = (p + item_size <= buf) ? 0 : (u32)(p - buf); + return found; +} + +/* + * Search an NT_GNU_PROPERTY_TYPE_0 for the property of 'pr_type'. + */ +static int find_property(struct file *file, unsigned long desc_size, + loff_t file_offset, u8 *buf, + u32 pr_type, u32 *property) +{ + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + u32 last_pr = 0; + + *property = 0; + buf_pos = 0; + + for (done = 0; done < desc_size; done += buf_pos) { + read_size = desc_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) + return (ret < 0) ? ret : -EIO; + + ret = 0; + found = scan(buf, read_size, sizeof(struct gnu_property), + test_property, next_property, + &last_pr, pr_type, &buf_pos); + + if ((!buf_pos) || found) + break; + + file_offset += buf_pos - read_size; + } + + if (found) { + struct gnu_property *pr = + (struct gnu_property *)(buf + buf_pos); + + if (pr->pr_datasz == 4) { + u32 *max = (u32 *)(buf + read_size); + u32 *data = (u32 *)((u8 *)pr + sizeof(*pr)); + + if (data + 1 <= max) { + *property = *data; + } else { + file_offset += buf_pos - read_size; + file_offset += sizeof(*pr); + ret = kernel_read(file, property, 4, + &file_offset); + } + } + } + + return ret; +} + +/* + * Search a PT_NOTE segment for NT_GNU_PROPERTY_TYPE_0. + */ +static int find_note_type_0(struct file *file, loff_t file_offset, + unsigned long note_size, u32 align, + u32 pr_type, u32 *property) +{ + u8 *buf; + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + + buf = kmalloc(BUF_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + *property = 0; + buf_pos = 0; + + for (done = 0; done < note_size; done += buf_pos) { + read_size = note_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) { + ret = (ret < 0) ? ret : -EIO; + kfree(buf); + return ret; + } + + /* + * item_size = sizeof(struct elf_note) + elf_note.n_namesz. + * n_namesz is 4 for the note type we look for. + */ + ret = scan(buf, read_size, sizeof(struct elf_note) + 4, + test_note_type, next_note, + &align, NT_GNU_PROPERTY_TYPE_0, &buf_pos); + + file_offset += buf_pos - read_size; + + if (ret && !found) { + struct elf_note *n = + (struct elf_note *)(buf + buf_pos); + u64 start = round_up(sizeof(*n) + n->n_namesz, align); + u64 total = 0; + + if (check_add_overflow(start, (u64)n->n_descsz, &total)) { + ret = -EINVAL; + break; + } + total = round_up(total, align); + + ret = find_property(file, n->n_descsz, + file_offset + start, + buf, pr_type, property); + found++; + file_offset += total; + buf_pos += total; + } else if (!buf_pos || ret) { + ret = 0; + *property = 0; + break; + } + } + + kfree(buf); + return ret; +} + +/* + * Look at an ELF file's PT_NOTE segments, then NT_GNU_PROPERTY_TYPE_0, then + * the property of pr_type. + * + * Input: + * file: the file to search; + * phdr: the file's elf header; + * phnum: number of entries in phdr; + * pr_type: the property type. + * + * Output: + * The property found. + * + * Return: + * Zero or error. + */ +static int scan_segments_64(struct file *file, struct elf64_phdr *phdr, + int phnum, u32 pr_type, u32 *property) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 8)) + continue; + + /* + * Search the PT_NOTE segment for NT_GNU_PROPERTY_TYPE_0. + */ + err = find_note_type_0(file, phdr->p_offset, phdr->p_filesz, + phdr->p_align, pr_type, property); + if (err) + return err; + } + + return 0; +} + +static int scan_segments_32(struct file *file, struct elf32_phdr *phdr, + int phnum, u32 pr_type, u32 *property) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 4)) + continue; + + /* + * Search the PT_NOTE segment for NT_GNU_PROPERTY_TYPE_0. + */ + err = find_note_type_0(file, phdr->p_offset, phdr->p_filesz, + phdr->p_align, pr_type, property); + if (err) + return err; + } + + return 0; +} + +int get_gnu_property(void *ehdr_p, void *phdr_p, struct file *f, + u32 pr_type, u32 *property) +{ + struct elf64_hdr *ehdr64 = ehdr_p; + int err = 0; + + *property = 0; + + if (ehdr64->e_ident[EI_CLASS] == ELFCLASS64) { + struct elf64_phdr *phdr64 = phdr_p; + + err = scan_segments_64(f, phdr64, ehdr64->e_phnum, + pr_type, property); + if (err < 0) + goto out; + } else { + struct elf32_hdr *ehdr32 = ehdr_p; + + if (ehdr32->e_ident[EI_CLASS] == ELFCLASS32) { + struct elf32_phdr *phdr32 = phdr_p; + + err = scan_segments_32(f, phdr32, ehdr32->e_phnum, + pr_type, property); + if (err < 0) + goto out; + } + } + +out: + return err; +} diff --git a/include/linux/elf.h b/include/linux/elf.h index e3649b3e970e..c15febebe7f2 100644 --- a/include/linux/elf.h +++ b/include/linux/elf.h @@ -56,4 +56,16 @@ static inline int elf_coredump_extra_notes_write(struct coredump_params *cprm) { extern int elf_coredump_extra_notes_size(void); extern int elf_coredump_extra_notes_write(struct coredump_params *cprm); #endif + +#ifdef CONFIG_ARCH_USE_GNU_PROPERTY +extern int arch_setup_property(void *ehdr, void *phdr, struct file *f, + bool interp); +extern int get_gnu_property(void *ehdr_p, void *phdr_p, struct file *f, + u32 pr_type, u32 *feature); +#else +static inline int arch_setup_property(void *ehdr, void *phdr, struct file *f, + bool interp) { return 0; } +static inline int get_gnu_property(void *ehdr_p, void *phdr_p, struct file *f, + u32 pr_type, u32 *feature) { return 0; } +#endif #endif /* _LINUX_ELF_H */ diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index 34c02e4290fe..316177ce9e76 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -372,6 +372,7 @@ typedef struct elf64_shdr { #define NT_PRFPREG 2 #define NT_PRPSINFO 3 #define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5 #define NT_AUXV 6 /* * Note to userspace developers: size of NT_SIGINFO note may increase @@ -443,4 +444,17 @@ typedef struct elf64_note { Elf64_Word n_type; /* Content type */ } Elf64_Nhdr; +/* NT_GNU_PROPERTY_TYPE_0 header */ +struct gnu_property { + __u32 pr_type; + __u32 pr_datasz; +}; + +/* .note.gnu.property types */ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* Bits of GNU_PROPERTY_X86_FEATURE_1_AND */ +#define GNU_PROPERTY_X86_FEATURE_1_IBT (0x00000001) +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + #endif /* _UAPI_LINUX_ELF_H */ From patchwork Thu Jun 6 20:06:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68B9F6C5 for ; Thu, 6 Jun 2019 20:16:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56AEC28A6D for ; Thu, 6 Jun 2019 20:16:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A89A28AA8; Thu, 6 Jun 2019 20:16:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF84328A6D for ; Thu, 6 Jun 2019 20:16:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C42056B02B7; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BCF786B02B9; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A930F6B02BA; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 6CA306B02B7 for ; Thu, 6 Jun 2019 16:15:40 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id j21so2064641pff.12 for ; Thu, 06 Jun 2019 13:15:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=3rc5WSC9lu5I3YPet9/X+BLe3GdVT7Iar3wD/bHVsGQ=; b=fq4MS7uaQZuJNPGO4vlBbDVvMjH9EKyCjLxA2VoCC4GmkgaiOgC604orWCoIIhSVn4 SuMtJcs+/27G6aGb7Se2yGFK9hb9aa8uucmIxuBfgiW4mOSSbUgqWGpVp+aYx+tASgA8 RXDeCmU5jljo/3rbWBiFGLbLMwYBg7TzHJ0qYdNKAcDI2C8LGE5fwjsuwQwHSW4gVUqm VbWaIqQ56bXO38jl1F8K+4FYQdos5MRf4SiNGTuDz2uHwAq/JzRcpAGex5mNG8fgaf57 kFUf3bQqysOLTlN9mEKMGg2oo9a0XbGFp93DLPcLqnTQyTw9xpfLzSsZRQYvGRXk/Yg5 UA1w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAULCqqvMiWoQTnKNfJCzTqBsxRVYDWypjg5RxInZ9FcsAmx6Abk LUkvfwI4wMjQaFoyNV9Td+H5CsQJDOa7ffbe5O0BsinIeYd1O6oB4SVsC2icarWQxe5eOPz4TId mO0BsAtW67xdVu43BQBWegXEWvcxHIQV+7bJJ30hOEKazSqcJGThdZWut9wBOQZG88A== X-Received: by 2002:aa7:8dcd:: with SMTP id j13mr53474952pfr.107.1559852140112; Thu, 06 Jun 2019 13:15:40 -0700 (PDT) X-Google-Smtp-Source: APXvYqyfKyIQKg6elaj5Q+zcn3QyzweClOISOYzY5CgYjtSUBoCmghpk3Vnx1bwur2IQyTkJUEej X-Received: by 2002:aa7:8dcd:: with SMTP id j13mr53474889pfr.107.1559852139136; Thu, 06 Jun 2019 13:15:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852139; cv=none; d=google.com; s=arc-20160816; b=YE6gbmQ4MgB1r4yzwpb+3T3bJLMxPwUEa9gzFjPzN2D2abXZaeYFCN849LYA95+bdo TjJHYeic0/pVr7VT7zf7+Eqr2pqKUFUv9UZg122okvdmjSwiW49Vb10BYMG5QSJGCYA+ M74NCW2MKPEuP+Or+HMLrkxGvDxzhNZC4VHh04PbqVY8MdeFg9wG08vYE7hQ9vH4lxEI hjUNnnq3s63DhNS4IkNHiMRKW3DJLzlbPdH2aFXdMYi85865+amL+kMScDDIKMGO6Qnr qm6uR6HrYZ/1sSwaF8A9TncDAwn3LekLPeFkbNzJVr5FMq8mvnq1IWVttinoUwQuDdxu UZLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=3rc5WSC9lu5I3YPet9/X+BLe3GdVT7Iar3wD/bHVsGQ=; b=PFHZp2oLQoSCZx0x/MeWvt9p3xJzjLXuE8I5eLVPTVwMsBz3LRMfcpzQ66R86Dyvit FTOj8FR/Ax3z/TqIdCUrhFGVU9YVEWbEpmWn08w5f0jfJ70tUuD3zwsJ91oQuJFixMYr 80CtGgubuA75Xov7M5qVg2bP1pe+mSuu1JqnIpC+JinapaCp4Q6oocO+fJPs6pUSGvfH Q/AgrU++16nXbwHp/lyP/fSHyfPn9Mr+8GTVykIrFd0YcKovNUAtltS5SNO1vSNST25U cYIRofZXeNQgkQRk6NcAGUv/gyPfZLqv5e1irS0q+fTb5d1KxQ83V0QiT0yC2u8tOFYU Y8Xw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:39 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:38 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:37 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 23/27] x86/cet/shstk: ELF header parsing of Shadow Stack Date: Thu, 6 Jun 2019 13:06:42 -0700 Message-Id: <20190606200646.3951-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 1 + arch/x86/kernel/process_64.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1664918c2c1c..df8b57de75b2 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1928,6 +1928,7 @@ config X86_INTEL_SHADOW_STACK_USER select ARCH_USES_HIGH_VMA_FLAGS select X86_INTEL_CET select ARCH_HAS_SHSTK + select ARCH_USE_GNU_PROPERTY ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 250e4c4ac6d9..c51df5b4e116 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -828,3 +828,27 @@ unsigned long KSTK_ESP(struct task_struct *task) { return task_pt_regs(task)->sp; } + +#ifdef CONFIG_ARCH_USE_GNU_PROPERTY +int arch_setup_property(void *ehdr, void *phdr, struct file *f, bool inter) +{ + int r; + uint32_t property; + + r = get_gnu_property(ehdr, phdr, f, GNU_PROPERTY_X86_FEATURE_1_AND, + &property); + + memset(¤t->thread.cet, 0, sizeof(struct cet_status)); + + if (r) + return r; + + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) { + if (property & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + r = cet_setup_shstk(); + if (r < 0) + return r; + } + return r; +} +#endif From patchwork Thu Jun 6 20:06:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980411 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F5F76C5 for ; Thu, 6 Jun 2019 20:16:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F1BB928A6D for ; Thu, 6 Jun 2019 20:16:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E5B0328AA7; Thu, 6 Jun 2019 20:16:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4999228A6D for ; Thu, 6 Jun 2019 20:16:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDED16B02BB; Thu, 6 Jun 2019 16:15:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C695E6B02BD; Thu, 6 Jun 2019 16:15:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B09376B02BC; Thu, 6 Jun 2019 16:15:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 6B0366B02B9 for ; Thu, 6 Jun 2019 16:15:42 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id d2so2144957pla.18 for ; Thu, 06 Jun 2019 13:15:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=93Gwv6XnAt1YbfZeHX2fnzvAW5eh83qpiHtLTKRMHGE=; b=m27vzOtDtOFADDuuy52108UxlQfykoxtaSN6/TLD3B0h/plAXeVCzP8QxReeGa287D BOnA/+b6NpGNpolXUdB5p7IxibvdqT9/w49J+/deICgN38JzV1ibEdTYAPyLlef80f0s FvShX108JNlt4nwwF3BZJ2mcjxgK6mFW2LHRLzplcovO+bL+3464ntb3clEpTJ0w5Kfl 5IDXjExFLgIb1HyWplvxiyHiFfZNCSWbtDjdkKH76LsQJPvzh10iyRIOnrDPK8QqWv0F EHk5jIbjS7+0D8SM/93uYPSj6pUhjS2naHA8/ghPXpYR4ue2jRD2I4ARFJNeikxavEVw 3m3A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAWFq2tCC62JYKAkazX121L3NRZ7zgwzNbdEQNwDHy9wN3bOLmaX J7HrC28JF6FPhWkRqjxbSoLTzVUiQHXDtw3Y84cp7PQFP3N7l67+2w/A+B09CKgaJBM2EMQbQdN ZHJNYS1ppzMyU3yiQrFw39QkqygiBrDbckULQ05FnLnADRtcdawgR157PqVpTG3o0BQ== X-Received: by 2002:a62:7552:: with SMTP id q79mr34939971pfc.71.1559852142032; Thu, 06 Jun 2019 13:15:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqyyfMgMUsp/XTMX9yjOEsC6l0q8LVvYp1H9+muASRZvqVaRIIhtMePigwDYJLA1jXNGjB1z X-Received: by 2002:a62:7552:: with SMTP id q79mr34939866pfc.71.1559852140520; Thu, 06 Jun 2019 13:15:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852140; cv=none; d=google.com; s=arc-20160816; b=MhX1xgA8CQ3cjmnE6q66f6D8fWe+n6Ia8UlOaETtNM433FLRvV60OXUx5GB0hxXKMO t03M8eiet0sRV2Ac6WMrI4ZMI1hjHlgz1dzpvVKkAhI027EAHfaQRCZtPZWmVyQqj0fT Fc4HyKlK0ltlEytFxbuaT+OPPC0XoJGnTWL84ut5Pb80HKfkn399vqdpwfkI9UaZOum3 /oZJQBQmtghVUhWWxFnDNLa0FVLm9jp7BCMQjNJXCL7KPL6AearL+xGMGjggyzOe+6vi 5yssJ+zeHJSWXWfusXn9rkba6XUFsSu1nMyt6Qd3oztkwN2msKPT/p6S8JfxBFju/O66 MoMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=93Gwv6XnAt1YbfZeHX2fnzvAW5eh83qpiHtLTKRMHGE=; b=myralCXL5KVr/PInZewvNq4VM07tGLI8226E60M8uRY0Dd1s+jer8ywvXVxHamCo8i tmz7aNc4LVymEqKcYruqv/jv53AhqgSXDDHY5vRUFIPFaK+B1EY/1IiLjwrzcVimsHdK +bU2fPu6K0eBoKYq8A/B0uV+JK96ek43s8mGP4DPJaaJhGkX646QOAM8RO0IZQFS/vFz OsM0N1vJcm88EPvkMaFvvUChmZSNZA8Jd5oMxSDJYOci4Z1Ue4dGiUG0xi4O7lHj0ke7 Mxsy1xcjurXORexND+NgyTPuUXQcIlQBBGw7E9taXhPWS4ZTYA8W+qSGMTZjdeGpg/cA QkNg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:40 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:40 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:38 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 24/27] x86/cet/shstk: Handle thread shadow stack Date: Thu, 6 Jun 2019 13:06:43 -0700 Message-Id: <20190606200646.3951-25-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The shadow stack for clone/fork is handled as the following: (1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child. It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK. (2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK. (3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret. This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches. A 64-bit SHSTK has a fixed size of RLIMIT_STACK. A compat-mode thread SHSTK has a fixed size of 1/4 RLIMIT_STACK. This allows more threads to share a 32-bit address space. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 41 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 +++++ 5 files changed, 54 insertions(+) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 422ccb8adbb7..52c506a68848 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -19,12 +19,14 @@ struct cet_status { #ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk(void); +int cet_setup_thread_shstk(struct task_struct *p); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(bool ia32, struct sc_ext *sc); int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); #else static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(bool ia32, struct sc_ext *sc) { return -EINVAL; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 9024236693d2..a9a768529540 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include #include extern atomic64_t last_mm_ctx_id; @@ -228,6 +229,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index b247cd15c1e2..9ef1af617d38 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -151,6 +151,47 @@ int cet_setup_shstk(void) return 0; } +int cet_setup_thread_shstk(struct task_struct *tsk) +{ + unsigned long addr, size; + struct cet_user_state *state; + + if (!current->thread.cet.shstk_enabled) + return 0; + + state = get_xsave_addr(&tsk->thread.fpu.state.xsave, + XFEATURE_CET_USER); + + if (!state) + return -EINVAL; + + size = rlimit(RLIMIT_STACK); + + /* + * Compat-mode pthreads share a limited address space. + * If each function call takes an average of four slots + * stack space, we need 1/4 of stack size for shadow stack. + */ + if (in_compat_syscall()) + size /= 4; + + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + if (addr >= TASK_SIZE_MAX) { + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + tsk->thread.cet.shstk_enabled = 0; + return -ENOMEM; + } + + fpu__prepare_write(&tsk->thread.fpu); + state->user_ssp = (u64)(addr + size - sizeof(u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; + return 0; +} + void cet_disable_shstk(void) { u64 r; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index a4deb79b1089..58b1c52b38b5 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -130,6 +130,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + cet_disable_free_shstk(tsk); fpu__drop(fpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index c51df5b4e116..5fa0d9ab18f1 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -421,6 +421,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp, if (sp) childregs->sp = sp; + /* Allocate a new shadow stack for pthread */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM) { + err = cet_setup_thread_shstk(p); + if (err) + goto out; + } + err = -ENOMEM; if (unlikely(test_tsk_thread_flag(me, TIF_IO_BITMAP))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr, From patchwork Thu Jun 6 20:06:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980413 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 450A16C5 for ; Thu, 6 Jun 2019 20:16:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 30FF428A6D for ; Thu, 6 Jun 2019 20:16:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 22E6028AA7; Thu, 6 Jun 2019 20:16:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C05B728A6D for ; Thu, 6 Jun 2019 20:16:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CDDE6B02B9; Thu, 6 Jun 2019 16:15:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 231456B02BD; Thu, 6 Jun 2019 16:15:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D37B6B02BE; Thu, 6 Jun 2019 16:15:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id C22BE6B02B9 for ; Thu, 6 Jun 2019 16:15:42 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id d7so2596039pfq.15 for ; Thu, 06 Jun 2019 13:15:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=03A9LmtVlu2jdr4RND01DjQZ3G+si2RYtelfS6jbPdM=; b=Ua1I0mrUgKCEhpKiaiFxwEAsLa4b0cSEk7evwDy1LATeOT9nr4sgSe8FSLe5SOkjrl 9z4MwurCUpyqMBcK2gPJwpk+/E6CMRjij/wgEIiXE3aIUVX/03jyqO6dHs+nwn/xq+Da 1w62q3iQk2tZR8LFDel/Tip3dBTND8TVSXlkwTOvMfMEOwe+i9ZDOI/P01ulWMuJ7fHN YmEDIKbIAFERYYLrRY9pfnjE99J7mVr0bzaTuEvHrz2LXE4hZlgvgCO8IBq8+MStUvCI +LVJVHnR0728GJ1/0wmsANdzoRIU9WuKnvg8TrYIMnpxuSPaWlPhHzebMwV4AJSB5TJ8 wD8Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAVKztJzc/iljJH32TgKaHYMtIQqGe1M7y8LdcpR6Wat6DRPi6sx bqxsuxinjxZSiQW0vrUtRofkcgj9nn/gklQTWzvwOMNv0z4a6vHjmpqhIptF9+ZZniVzAdf3nat 3dnlM3Z+J6ygwAyjIBYvR3UTELxN8owiHn4zLvYeWx2hmu1bTZ5nPCH7hqah0xVo7gQ== X-Received: by 2002:a17:90a:ba81:: with SMTP id t1mr1600472pjr.139.1559852142440; Thu, 06 Jun 2019 13:15:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqxwv+Ewhzd+hSGQMrnisB2CGLu9sb2IEzC2RWa3suaipKIs1Z1yj12Vr++Cbu8Jh+MkvFl7 X-Received: by 2002:a17:90a:ba81:: with SMTP id t1mr1600417pjr.139.1559852141740; Thu, 06 Jun 2019 13:15:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852141; cv=none; d=google.com; s=arc-20160816; b=hY9CdFPXwt0AsnYfD5cCwdukaR1PuF9in14UAr/qDBGJUM/LJpUrN2zfLEgV8tF7pr HFyerYrmB3rCO21P8kxsT2xQ6Q0/BFnsPNmYErdT+53j486YYZ2SZL3FVd3o5u5JAH16 PQm25YIYnMpt8LD43LwYUr23sotgzjYSPyFwz4H6gg3fOAsbZ2B266sWu2YSIoJ+iX2a c2s3SilRwp9NaaIrE47Sz0KdR0dPpen9NXdB2lE0MrBVN/yg5GF2UyLFp49wUa4IFTd2 2cQxKOrvzhYG13xAKAGOKgEKL69e0I1xrCN4lxRGc6qQ4BWJzh6Vjv1j1qjTNVpUl7sH gv2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=03A9LmtVlu2jdr4RND01DjQZ3G+si2RYtelfS6jbPdM=; b=oo9IpXnfZmoWwVa+47QVB+z6QyZwI71aZPOJTP8YD1esp2NCTC9X6Sytq9EsIxz/GO CDtx1SaTChH0GvP3Z8aES6O54xjglzQgdqIeb45BqXmVu77M6w3OW332J/bpSo0Z0B24 KkBOB7eLNjzj+VDcE2YNzcOBxOuroOAIMuw4Jrqicaz0BUVfEAIbQf//x00UVa3YWvak od3LVdcCiF30Tu1BDi1vzX7XNkLd6tnKSVMz+V0+w05v8b2I03Hfo2QqbT2sZnEQOlWY fyo7EUOsyDV7ksUJ34OuOhnhk5DtpNvVm1ziEGS1lUE7SHSwsRiwIFPqn8PIh7X9uSS0 /93w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:41 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:41 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 25/27] mm/mmap: Add Shadow stack pages to memory accounting Date: Thu, 6 Jun 2019 13:06:44 -0700 Message-Id: <20190606200646.3951-26-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add shadow stack pages to memory accounting. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index b1a921c0de63..3b643ace2c49 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1703,6 +1703,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) if (file && is_file_hugepages(file)) return 0; + if (arch_copy_pte_mapping(vm_flags)) + return 1; + return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE; } @@ -3319,6 +3322,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; + else if (arch_copy_pte_mapping(flags)) + mm->data_vm += npages; } static vm_fault_t special_mapping_fault(struct vm_fault *vmf); From patchwork Thu Jun 6 20:06:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980415 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D394E1515 for ; Thu, 6 Jun 2019 20:16:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C29E528A6D for ; Thu, 6 Jun 2019 20:16:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B5A9B28AA7; Thu, 6 Jun 2019 20:16:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29E8628A6D for ; Thu, 6 Jun 2019 20:16:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 479246B02BD; Thu, 6 Jun 2019 16:15:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 42D556B02BF; Thu, 6 Jun 2019 16:15:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 207F46B02C0; Thu, 6 Jun 2019 16:15:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id D6A606B02BD for ; Thu, 6 Jun 2019 16:15:44 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id 145so2592131pfv.18 for ; Thu, 06 Jun 2019 13:15:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=aXgEkBRX3/eV+3WwLvh+MpC5kBqOIXoZ+F2d27hK+18=; b=jPOe2n/al4JSOR/RpWhP3K2xbHlsRgAre7050qRpc42TUclrpBOyyYL5kLX83Eiz19 qRzB1iso7O3YezXV0wso+zj1xBQVolAA6JIn8vLLIUjV7Fmz2Xq/ZN06dk14EF55xlw9 rLiqITSj0HT1sy+ye6s/4PPLGzITrvp3A/Rhmli31AA7nhbFyZFvIFkPJ5xfbKUlRJNZ J38rfaYPZn+mpMZU3tJP+aT9caYg3off1KIARxBmxb1Jwn/+56nC0meDjgOLbujBjucT HGRsip6swC8KrXeCb/X1QifR9xRXyrHfohG615vBaR1Dl8Zrj5xbXw8zI+khT+JyOAPF 569A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAW8qTiIa2gOuSwmD5ANsxc5+eddqR4E3+7Go5ZwGPr4/4XYZEhG U12cRQwbUUIKN873E5AEseL5L9rw7/meJIIF8tVAVFQgMeiOrHe0cnJY+9Vlv70U0yBC5rUC6e7 wjUPR7W0trDbjKpbSIQAswdIJ2V2MeScxZBsxTABzSHuFNRreuieGsGILacC52aCz/g== X-Received: by 2002:a62:7656:: with SMTP id r83mr30872202pfc.56.1559852144536; Thu, 06 Jun 2019 13:15:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqzTYPOqMoCHOsty0W6NEZxRvEGAh7g7oS2zlBDbLbnq7Kvye+unQVydObwW00dgMGbet1zT X-Received: by 2002:a62:7656:: with SMTP id r83mr30872102pfc.56.1559852143118; Thu, 06 Jun 2019 13:15:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852143; cv=none; d=google.com; s=arc-20160816; b=R8sU+lDaZcczuy+zpoWu2sFAYw3LC+0rOW7pK87/Fz0YFKgJw/tBzoT3l+v573sHAs 0iqI2hRqS5EqmwTri5PvWvUpspUlqpRE30NH/BGlPbWF55jRV8aQXVRWDnBpEOuS4k11 7edrJAiDdJ0uMVh7dCoNEyUYJBbOKLh61tOeLd2xI+rI7LwoQ/G+M9yJN9N3Qy8VVlJn z+utTCkw0L1FseuNRAlxwjV0OCtwK+/tB4IkXepYSNP9wgcGgqjLkZzi9UR62AOVC1TM Jc+nyIfdtuQZkPpPPhQkoasyx82v+kLASTb+ICMLYQaxzacxIXPAbrrsFzIMbbip1yKF UHPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=aXgEkBRX3/eV+3WwLvh+MpC5kBqOIXoZ+F2d27hK+18=; b=MZkkK8kMl8ELGe96hXzBhYoH5ucFZ2DVa07KfNb+cNVvoZjJVbYgHonlqbG6ONidHB sYOxxyVIMs851OmVmnJYpb52NHK5iKETys+Kv7ZfHF0Oe38KhTGGC2oRIvMhSrZjgGrX /k07EvYdYLC4L5A7XQKHkMpyds3ms8aYzbBA+g71MH5BpKn8uox6fltvtBiZZqQ6FUhq JxmqzBVtITmKJCIrtkqDl/2WFOvqFMuDKR4Yb+K3xts//7Ti0QGiiuEq+bXsfhirUvvl uCao4i0RBp2y/EmZPoKDM3JkaHue8SJJGJTY+MdSKPDKuZY9tr9LK3hMYLjSBvELP5xz OwZw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:43 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:42 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 26/27] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Date: Thu, 6 Jun 2019 13:06:45 -0700 Message-Id: <20190606200646.3951-27-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) Return CET feature status. The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information: *addr = SHSTK/IBT status *(addr + 1) = SHSTK base address *(addr + 2) = SHSTK size arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) Disable CET features specified in 'features'. Return -EPERM if CET is locked. arch_prctl(ARCH_X86_CET_LOCK) Lock in CET feature. arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) Allocate a new SHSTK. The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK. There is no CET enabling arch_prctl function. By design, CET is enabled automatically if the binary and the system can support it. The parameters passed are always unsigned 64-bit. When an ia32 application passing pointers, it should only use the lower 32 bits. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 29 +++++++++++ arch/x86/kernel/cet_prctl.c | 85 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 4 +- 6 files changed, 127 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cet_prctl.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 52c506a68848..2df357dffd24 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -14,19 +14,24 @@ struct sc_ext; struct cet_status { unsigned long shstk_base; unsigned long shstk_size; + unsigned int locked:1; unsigned int shstk_enabled:1; }; #ifdef CONFIG_X86_INTEL_CET +int prctl_cet(int option, unsigned long arg2); int cet_setup_shstk(void); int cet_setup_thread_shstk(struct task_struct *p); +int cet_alloc_shstk(unsigned long *arg); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(bool ia32, struct sc_ext *sc); int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); #else +static inline int prctl_cet(int option, unsigned long arg2) { return -EINVAL; } static inline int cet_setup_shstk(void) { return -EINVAL; } static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } +static inline int cet_alloc_shstk(unsigned long *arg) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(bool ia32, struct sc_ext *sc) { return -EINVAL; } diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 5a6aac9fa41f..d962f0ec9ccf 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -14,4 +14,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_X86_CET_STATUS 0x3001 +#define ARCH_X86_CET_DISABLE 0x3002 +#define ARCH_X86_CET_LOCK 0x3003 +#define ARCH_X86_CET_ALLOC_SHSTK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 584ed7e9a599..d908c95306fc 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -140,7 +140,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o ### # 64 bit specific files diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 9ef1af617d38..0004333f8373 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -127,6 +127,35 @@ static int create_rstor_token(bool ia32, unsigned long ssp, return 0; } +int cet_alloc_shstk(unsigned long *arg) +{ + unsigned long len = *arg; + unsigned long addr; + unsigned long token; + unsigned long ssp; + + addr = do_mmap_locked(0, len, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + if (addr >= TASK_SIZE_MAX) + return -ENOMEM; + + /* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + + if (!in_ia32_syscall()) + token |= TOKEN_MODE_64; + ssp -= 8; + + if (write_user_shstk_64(ssp, token)) { + vm_munmap(addr, len); + return -EINVAL; + } + + *arg = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c new file mode 100644 index 000000000000..9c9d4262b07e --- /dev/null +++ b/arch/x86/kernel/cet_prctl.c @@ -0,0 +1,85 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* See Documentation/x86/intel_cet.rst. */ + +static int handle_get_status(unsigned long arg2) +{ + unsigned int features = 0; + unsigned long shstk_base, shstk_size; + unsigned long buf[3]; + + if (current->thread.cet.shstk_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + + shstk_base = current->thread.cet.shstk_base; + shstk_size = current->thread.cet.shstk_size; + + buf[0] = (unsigned long)features; + buf[1] = shstk_base; + buf[2] = shstk_size; + return copy_to_user((unsigned long __user *)arg2, buf, + sizeof(buf)); +} + +static int handle_alloc_shstk(unsigned long arg2) +{ + int err = 0; + unsigned long arg; + unsigned long addr = 0; + unsigned long size = 0; + + if (get_user(arg, (unsigned long __user *)arg2)) + return -EFAULT; + + size = arg; + err = cet_alloc_shstk(&arg); + if (err) + return err; + + addr = arg; + if (put_user(addr, (unsigned long __user *)arg2)) { + vm_munmap(addr, size); + return -EFAULT; + } + + return 0; +} + +int prctl_cet(int option, unsigned long arg2) +{ + if (!cpu_x86_cet_enabled()) + return -EINVAL; + + switch (option) { + case ARCH_X86_CET_STATUS: + return handle_get_status(arg2); + + case ARCH_X86_CET_DISABLE: + if (current->thread.cet.locked) + return -EPERM; + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + cet_disable_free_shstk(current); + + return 0; + + case ARCH_X86_CET_LOCK: + current->thread.cet.locked = 1; + return 0; + + case ARCH_X86_CET_ALLOC_SHSTK: + return handle_alloc_shstk(arg2); + + default: + return -EINVAL; + } +} diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 58b1c52b38b5..e0090f2790df 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -873,7 +873,7 @@ long do_arch_prctl_common(struct task_struct *task, int option, return get_cpuid_mode(); case ARCH_SET_CPUID: return set_cpuid_mode(task, cpuid_enabled); + default: + return prctl_cet(option, cpuid_enabled); } - - return -EINVAL; } From patchwork Thu Jun 6 20:06:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10980417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8958E1515 for ; Thu, 6 Jun 2019 20:16:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78A5628A6D for ; Thu, 6 Jun 2019 20:16:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6AB6B28AA7; Thu, 6 Jun 2019 20:16:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A54F728A6D for ; Thu, 6 Jun 2019 20:16:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42DF56B02BF; Thu, 6 Jun 2019 16:15:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3B5066B02C1; Thu, 6 Jun 2019 16:15:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 258B26B02C2; Thu, 6 Jun 2019 16:15:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id CED0C6B02BF for ; Thu, 6 Jun 2019 16:15:45 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id bc12so2182171plb.0 for ; Thu, 06 Jun 2019 13:15:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=d3zi1pPwT2xl7ig4a9Cv4Tbt+P/hcGeGhr6wvfa2F+O9o3iFzvJdAKz2NfDzeYhRNF MkrtrsWtq2ZH4begfkCvHPB+WbFkhvMHj6LUGN/aAHhmU6f8qGVZvY19fnvxnopQa8Lk DluXdINzhDI4JPgDQ47JM0fM8NerEeI3qoOQX6aY/NyzjjdXN3f2Zn2eROuTIXivJU4E aJMXiT0wgn864Ma8pqTDXpPkVzJ/1KeTaZhUtjfZLQSD+Ic4Oe6lSEUNvr3ARMeWuSZ2 8LTsxOy0iGlOsSo/FrxSNm7X5BOwjCfGhekEdkVIfoVeRFnCGC01qmAItehi1J3KxMfh Oc0Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAX0C+gzOiLj5yCZL3KXDXs+t3mHYci7nSOQr5Dd2GE0tZomLS+e I3ce9pVw53xZUO1s0ZLQBXCkmmlcKBplllqNvQzgpZx8Txz/B7M0h+GjO1F/QHLjxIxDvAnAZck 6Z8zHMsl9+Idj4Gm8OOcaoFzp63HZLhaZ/fkEsjheFoP7t+yBSCSkVzVoWh9XWdDivQ== X-Received: by 2002:aa7:825a:: with SMTP id e26mr55750904pfn.255.1559852145485; Thu, 06 Jun 2019 13:15:45 -0700 (PDT) X-Google-Smtp-Source: APXvYqywDnhjwWERP2Qxl78eWfQTqlt9azNRDSqnH+MfOJsI7fm8oSad3iqb53xFKHN/FJtjXohd X-Received: by 2002:aa7:825a:: with SMTP id e26mr55750827pfn.255.1559852144393; Thu, 06 Jun 2019 13:15:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559852144; cv=none; d=google.com; s=arc-20160816; b=zJaHs5jSPRukiP87rNKIJJdw7dkY9vSKXpLzo49DGasSKaZynA4Op6+5CIh5KAwtVg AOY6yiBTlcp9hAxNmC6vRPILQV0OcO1OAfxzipLtESaeAaijdziwqD+D2KYS/fR3E2PQ PAmPvsmBRUtEr2wXCE8sGSjm//XxeOpX5rFCoUyv1YgCcv35qC8VjVGqrEkAvsPLkeUa uOC5RAzNnltdfsbBHWVTTQM9iQy2Qj57Dv48Odu8kZXPMf+J8/2jtl7/px8QBeFwklcf Bz0ZiH7QFD2WmpLczzHAX2JHx/iB0/5ncS2gWgz409TctT5oWmWUChPAoQR8BeDBs9+R VcRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=0rO0Ix3cS0s2YspOCLi61vWmj6rb1+FtZvsX4Eoo8Mu1908vOZNNzql+Hi0OH+q9wJ OFWJWvrE/7kYBf6F4p0SCCyVjATxBl8t0hnVZX+7ZdLyuvUHlPJvErwPFN1MDZnWNLge yTcRE6OYGZZclXRUG8rklEG+arDG11Z3aMZ0xkLXPrpDvFHeKOHCZ3+xGl7stpGEiRVq hJBoy1q6gtgPpAt6r8NVYXPHPg9/UJVJ5MN/1o2ZwrfRfMMdki1y6fK4eQDDC0HXRSog euplOc/YprWxjyvxNTH0Y0CyDRY0RrVs9Gn79rKk3Vmdk9cZAi3mjxrK08HrfnvaMMI9 Ze5g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id r142si2785300pfc.219.2019.06.06.13.15.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 13:15:44 -0700 (PDT) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Jun 2019 13:15:43 -0700 X-ExtLoop1: 1 Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga002.jf.intel.com with ESMTP; 06 Jun 2019 13:15:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v7 27/27] x86/cet/shstk: Add Shadow Stack instructions to opcode map Date: Thu, 6 Jun 2019 13:06:46 -0700 Message-Id: <20190606200646.3951-28-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190606200646.3951-1-yu-cheng.yu@intel.com> References: <20190606200646.3951-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the following shadow stack management instructions. INCSSP: Increment shadow stack pointer by the steps specified. RDSSP: Read SSP register into a GPR. SAVEPREVSSP: Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack. RSTORSSP: Restore from a "restore token" pointed by a GPR to SSP. WRSS: Write to kernel-mode shadow stack (kernel-mode instruction). WRUSS: Write to user-mode shadow stack (kernel-mode instruction). SETSSBSY: Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR. CLRSSBSY: Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable