From patchwork Mon Nov 19 21:47:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689515 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B82A14BD for ; Mon, 19 Nov 2018 21:54:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0D8C2A583 for ; Mon, 19 Nov 2018 21:54:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E4BE52A6EC; Mon, 19 Nov 2018 21:54:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB1C22A583 for ; Mon, 19 Nov 2018 21:54:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D33B26B1C87; Mon, 19 Nov 2018 16:54:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CBB6B6B1C89; Mon, 19 Nov 2018 16:54:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1AAC6B1C8A; Mon, 19 Nov 2018 16:54:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 6691E6B1C87 for ; Mon, 19 Nov 2018 16:54:14 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id d23so19890693plj.22 for ; Mon, 19 Nov 2018 13:54:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=vucfqwdq4c6gVa0lT7yg4ZnslOTMLmS9YvYL/tK/39g=; b=UWw2qyJXt5tAdKgwpq7SdERoiY/FlT2k4w9orZvbGKlRI4tfalRG0MSQocGM+zCLDz tU7mZZrxKlH5Dgzz6rVR/LWZpT+YNtvHOvoyTeHtWOjR7d1qWqSWLXfcqj3IA9B5dvMe MGt8E3l9GNv8fWwXszytBrUlmomY4vJJ0uoAmQUOUbSd+JgeDRg3YWOu80nQpFE8byas mFWU61q3ArYwrCjGjdZ4J5spvqNllkH7Zo1lCsbXVjY7aOvGyY9lGHWGCXWJQjI5ovvp Fipg9VYvy6WFW2UeGWBqrsVYcY6xrv49UPPyVX3VNes3LmdrVrDOxb0qN7DlCGQOXNzF Jwlg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gL+rl1rB11s1U6wOq4xkWtzPuh8rewuxrBHUJwF49EL1PL7whre Ts0WyrIyr7An/iTOWIW8KIitiS7eWtR7FdrNqHzjVHTrLDzDRXLoQQ9Zuf1QbWVCB3ilG/IreJO zGZ6NN5nQ8sybv4QBA3WPvc9sJ9pfirHGoFAZfSzHPo4mLvfWDrfG+fp8YfGLLij3Sg== X-Received: by 2002:a17:902:f24:: with SMTP id 33-v6mr24814707ply.228.1542664454026; Mon, 19 Nov 2018 13:54:14 -0800 (PST) X-Google-Smtp-Source: AJdET5cYSThLPz/ngnJIkh9e8x3ir+ZQNpR8RPnexZJUhPzvR83bYNhCU/t8TbdJWMa+PDdnTxJZ X-Received: by 2002:a17:902:f24:: with SMTP id 33-v6mr24814625ply.228.1542664452453; Mon, 19 Nov 2018 13:54:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664452; cv=none; d=google.com; s=arc-20160816; b=h5Zst9luW12AdsZnbYABwQma5+KgBT51Yv5DE/1oPg/OYCQ+1LgMnLFhlDZZJ4GO45 7bXhJFI2FpofLaMmznVfvNOZwRlEaQYUghM4N6l3vU+mIa+gIJJROYjddt7b+4SE4jfc ujPsI9qG3E+Jr1wx9Y7dxBWRJpXA5Yc9Mwqq39crdDgYGN7/wC3qhwWNA4zlA2TvCsVN 1308RI5oQH3mlbHZ22eonVzpeB4W4DUbWnHydqf53kYZiuDFMz5AZF+VuG78nshP1cK/ 1Od2cJ9hqsLzhJhgbRSrkaDRf3e2V/rWOn/UAMUIlklnaL4DDt0QfTI6AGb1XCgK2wS4 NpLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=vucfqwdq4c6gVa0lT7yg4ZnslOTMLmS9YvYL/tK/39g=; b=UG3I+x9tY1tZ6m9DYjMzfG72E35/9Kicyq/GVhBi/UFeAeDGu4fWuTzjQMl3a+f7nS 8rCemljaURnQS4o5Sci9pDa949catLo4OFyKanr/+zls5QKmV4N3hbDaOTmZDHJUOZ5N UqbASQ7dUU2vu4JNc1D3/Bqj0SeorA/5hGUcwZkD12/cwgP8jNmDuCXbLy5cB87QJkgM p16m0pMQjCS9GzYeGOXEV9h2yVUG9fCJirK2fh/90QI7Hqv2zycXH3N9qQL3s9d7tWb8 yduI+YNhNsRHj4KBYkwUCz6C+BEQklKi/eW80crf6cQEp9GP3MbFXpuZszWQv9eR/Vlv J5SA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:12 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319808" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:10 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 01/26] Documentation/x86: Add CET description Date: Mon, 19 Nov 2018 13:47:44 -0800 Message-Id: <20181119214809.6086-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Explain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters. Signed-off-by: Yu-cheng Yu --- .../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 13 + Documentation/x86/intel_cet.rst | 268 ++++++++++++++++++ 4 files changed, 288 insertions(+) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 81d1d5a74728..68c827cd9fe4 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2792,6 +2792,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor. diff --git a/Documentation/index.rst b/Documentation/index.rst index c858c2e66e36..63864826dcd6 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -101,6 +101,7 @@ implementation. :maxdepth: 2 sh/index + x86/index Filesystem Documentation ------------------------ diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst new file mode 100644 index 000000000000..1741429bd256 --- /dev/null +++ b/Documentation/x86/index.rst @@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================= +X86 Documentation +======================= + +Control-flow Enforcement +======================== + +.. toctree:: + :maxdepth: 1 + + intel_cet diff --git a/Documentation/x86/intel_cet.rst b/Documentation/x86/intel_cet.rst new file mode 100644 index 000000000000..dac83bbf8a24 --- /dev/null +++ b/Documentation/x86/intel_cet.rst @@ -0,0 +1,268 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================= +Control-flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control-flow Enforcement Technology (CET) provides protection against +return/jump-oriented programming (ROP) attacks. It can be setup to +protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented in 64-bit mode; 32-bit +applications are supported in compatibility mode. + +CET introduces shadow stack (SHSTK) and indirect branch tracking +(IBT). SHSTK is a secondary stack allocated from memory and cannot +be directly modified by applications. When executing a CALL, the +processor pushes a copy of the return address to SHSTK. Upon +function return, the processor pops the SHSTK copy and compares it +to the one from the program stack. If the two copies differ, the +processor raises a control-protection exception. IBT verifies all +indirect CALL/JMP targets are intended as marked by the compiler +with 'ENDBR' opcodes (see CET instructions below). + +There are two kernel configuration options: + + INTEL_X86_SHADOW_STACK_USER, and + INTEL_X86_BRANCH_TRACKING_USER. + +To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or later +are required. To build a CET-enabled application, GLIBC v2.28 or +later is also required. + +There are two command-line options for disabling CET features: + + no_cet_shstk - disables SHSTK, and + no_cet_ibt - disables IBT. + +At run time, /proc/cpuinfo shows the availability of SHSTK and IBT. + +[2] CET assembly instructions +============================= + +RDSSP %r + Read the SHSTK pointer into %r. + +INCSSP %r + Unwind (increment) the SHSTK pointer (0 ~ 255) steps as indicated + in the operand register. The GLIBC longjmp uses INCSSP to unwind + the SHSTK until that matches the program stack. When it is + necessary to unwind beyond 255 steps, longjmp divides and repeats + the process. + +RSTORSSP (%r) + Switch to the SHSTK indicated in the 'restore token' pointed by + the operand register and replace the 'restore token' with a new + token to be saved (with SAVEPREVSSP) for the outgoing SHSTK. + +:: + + Before RSTORSSP + + Incoming SHSTK Current/Outgoing SHSTK + + |----------------------| |----------------------| + addr=x | | ssp-> | | + |----------------------| |----------------------| + (%r)-> | rstor_token=(x|Lg) | addr=y-8 | | + |----------------------| |----------------------| + + After RSTORSSP + + |----------------------| |----------------------| + | | | | + |----------------------| |----------------------| + ssp-> | rstor_token=(y|Bz|Lg)| addr=y-8 | | + |----------------------| |----------------------| + + note: + 1. Only valid addresses and restore tokens can be on the + user-mode SHSTK. + 2. A token is always of type u64 and must align to u64. + 3. The incoming SHSTK pointer in a rstor_token must point to + immediately above the token. + 4. 'Lg' is bit[0] of a rstor_token indicating a 64-bit SHSTK. + 5. 'Bz' is bit[1] of a rstor_token indicating the token is to + be used only for the next SAVEPREVSSP and invalid for the + RSTORSSP. + +SAVEPREVSSP + Store the SHSTK 'restore token' pointed by + (current_SHSTK_pointer + 8). + +:: + + After SAVEPREVSSP + + |----------------------| |----------------------| + ssp-> | | | | + |----------------------| |----------------------| + | rstor_token=(y|Bz|Lg)| addr=y-8 | rstor_token(y|Lg) | + |----------------------| |----------------------| + +WRUSS %r0, (%r1) + Write the value in %r0 to the SHSTK address pointed by (%r1). + This is a kernel-mode only instruction. + +ENDBR + The compiler inserts an ENDBR at all valid branch targets. Any + CALL/JMP to a target without an ENDBR triggers a control + protection fault. + +[3] Application Enabling +======================== + +An application's CET capability is marked in its ELF header and can +be verified from the following command output, in the +NT_GNU_PROPERTY_TYPE_0 field: + + readelf -n + +If an application supports CET and is statically linked, it will run +with CET protection. If the application needs any shared libraries, +the loader checks all dependencies and enables CET only when all +requirements are met. + +[4] Legacy Libraries +==================== + +GLIBC provides a few tunables for backward compatibility. + +GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-IBT + Turn off SHSTK/IBT for the current shell. + +GLIBC_TUNABLES=glibc.tune.x86_shstk= + This controls how dlopen() handles SHSTK legacy libraries: + on: continue with SHSTK enabled; + permissive: continue with SHSTK off. + +[5] CET system calls +==================== + +The following arch_prctl() system calls are added for CET: + +arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) + Return CET feature status. + + The parameter 'addr' is a pointer to a user buffer. + On returning to the caller, the kernel fills the following + information: + + *addr = SHSTK/IBT status + *(addr + 1) = SHSTK base address + *(addr + 2) = SHSTK size + +arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) + Disable SHSTK and/or IBT specified in 'features'. Return -EPERM + if CET is locked. + +arch_prctl(ARCH_X86_CET_LOCK) + Lock in CET feature. + +arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) + Allocate a new SHSTK and put a restore token at top. + + The parameter 'addr' is a pointer to a user buffer and indicates + the desired SHSTK size to allocate. On returning to the caller, + the kernel fills *addr with the base address of the new SHSTK. + +arch_prctl(ARCH_X86_CET_SET_LEGACY_BITMAP, unsigned long *addr) + Setup an IBT legacy code bitmap. + + The parameter 'addr' is a pointer to a user buffer that has the + following information: + + *addr = IBT bitmap base address + *(addr + 1) = IBT bitmap size + +Note: + There is no CET enabling arch_prctl function. By design, CET is + enabled automatically if the binary and the system can support it. + + The parameters passed are always unsigned 64-bit. When an ia32 + application passing pointers, it should only use the lower 32 bits. + +[6] The implementation of the SHSTK +=================================== + +SHSTK size +---------- + +A task's SHSTK is allocated from memory to a fixed size of +RLIMIT_STACK. A compat-mode thread's SHSTK size is 1/4 of +RLIMIT_STACK. The smaller 32-bit thread SHSTK allows more threads to +share a 32-bit address space. + +Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, a large SHSTK will cover the +condition that both the program stack and the sigaltstack run out. + +The kernel creates a restore token at the SHSTK restoring address and +verifies that token when restoring from the signal handler. + +Fork +---- + +The SHSTK's vma has VM_SHSTK flag set; its PTEs are required to be +read-only and dirty. When a SHSTK PTE is not present, RO, and dirty, +a SHSTK access triggers a page fault with an additional SHSTK bit set +in the page fault error code. + +When a task forks a child, its SHSTK PTEs are copied and both the +parent's and the child's SHSTK PTEs are cleared of the dirty bit. +Upon the next SHSTK access, the resulting SHSTK page fault is handled +by page copy/re-use. + +When a pthread child is created, the kernel allocates a new SHSTK for +the new thread. + +Setjmp/Longjmp +-------------- + +Longjmp unwinds SHSTK until it matches the program stack. + +Ucontext +-------- + +In GLIBC, getcontext/setcontext is implemented in similar way as +setjmp/longjmp. + +When makecontext creates a new ucontext, a new SHSTK is allocated for +that context with ARCH_X86_CET_ALLOC_SHSTK the syscall. The kernel +creates a restore token at the top of the new SHSTK and the user-mode +code switches to the new SHSTK with the RSTORSSP instruction. + +[7] The management of read-only & dirty PTEs for SHSTK +====================================================== + +A RO and dirty PTE exists in the following cases: + +(a) A page is modified and then shared with a fork()'ed child; +(b) A R/O page that has been COW'ed; +(c) A SHSTK page. + +The processor only checks the dirty bit for (c). To prevent the use +of non-SHSTK memory as SHSTK, we use a spare bit of the 64-bit PTE as +DIRTY_SW for (a) and (b) above. This results to the following PTE +settings: + +Modified PTE: (R/W + DIRTY_HW) +Modified and shared PTE: (R/O + DIRTY_SW) +R/O PTE, COW'ed: (R/O + DIRTY_SW) +SHSTK PTE: (R/O + DIRTY_HW) +SHSTK PTE, COW'ed: (R/O + DIRTY_HW) +SHSTK PTE, shared: (R/O + DIRTY_SW) + +Note that DIRTY_SW is only used in R/O PTEs but not R/W PTEs. + +[8] The implementation of IBT +============================= + +The kernel provides IBT support in mmap() of the legacy code bit map. +However, the management of the bitmap is done in the GLIBC or the +application. From patchwork Mon Nov 19 21:47:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689517 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAF5813BB for ; Mon, 19 Nov 2018 21:54:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D7A62A583 for ; Mon, 19 Nov 2018 21:54:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 915C42A6EC; Mon, 19 Nov 2018 21:54:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58EA42A583 for ; Mon, 19 Nov 2018 21:54:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D1B46B1C88; Mon, 19 Nov 2018 16:54:15 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 25AB26B1C8B; Mon, 19 Nov 2018 16:54:15 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 103296B1C88; Mon, 19 Nov 2018 16:54:15 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id A58C56B1C88 for ; Mon, 19 Nov 2018 16:54:14 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id v72so21614015pgb.10 for ; Mon, 19 Nov 2018 13:54:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=VpZY8ZLqTTgQFw0nBILIEIjNm6g5tzhmbHYHTpHvidg=; b=gP7A0gq3/mYHnOaxak4qw7axLx7wAeU2KMe/6BNwENThE+RO0tGXZjbKSZTkRQagWj BJ2Jn0Uu3onaCwMDgQ2veitalqgfQzXaP+QONc168A6JpoCmYRzBDdiyxSqHCeqKNy8j NoLecR715WgBK7XeGdSHF6ck4iiJMV8r/buDbc4xLaPLzmO7l5CU2DfbBMPP2jfbScTo lvMa2gq/S0QdwcBpreaL/dm8alU0SZTvUgOYot7jLyopxJbmrBATzPKbQkhdSaB8lGdC xPjW8baJeLViZqmjN+3kMwp66jyolE2n09XGtMjbBmMglpPVZtriwc0tVwh8Hqc79gUd qhVA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gL0vny1V1HjalLoL7+fxqxPmTsADN0E5dNC392vT3CRe814H98w f1RX++o7AZwrN200OZ3na59kHs7SWhVssCTBou9YI0ULIEOqC89BOKB+AogTy/1jlMcHgxlag/7 EucDZ6rMxk4l//NPwpkoG3g+qDkALRyZsiFpS6mZl58Uf2tcxwRDpGL6dG2G2fXv4xw== X-Received: by 2002:a63:b34f:: with SMTP id x15mr22065226pgt.243.1542664454311; Mon, 19 Nov 2018 13:54:14 -0800 (PST) X-Google-Smtp-Source: AJdET5dc+1A+KX+HIWKHkkoYahvaAHGnVObymmg3QGquEXEmq5qM5e69aoOjJ79O/Y9z8yXd8NrF X-Received: by 2002:a63:b34f:: with SMTP id x15mr22065182pgt.243.1542664453271; Mon, 19 Nov 2018 13:54:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664453; cv=none; d=google.com; s=arc-20160816; b=Y7AFHpYGA4WfLShUYa/F/O52jJf0n3Cr8Ys6iVD0METE3mNwUKjxDbrId9QcVSyCRz 0mrIhxKm/05sdHT8UWJkb8oKH9lK0DxRW0FveGMl4pLiIkUCrQLJVqjN5hj0gy6K2ohF QqORFhgYBin9NdPxsw0RoPTbp55jYHlKDolkerbdgCw0VSLRAIS0uJRJziQ5c1iDsC+B q+QPCavnq2to0hKlqmfPESDE+3B43BUuunS38PqYfXeCaecID1MJcfNR2+X3RmqnWnek GyXl4I74PLRhdKBcTBl2jU2GMHn0tnLvzrigs4V0U2lgZ858ttnJK1Dn/QBoq3PRpQcM 2vww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=VpZY8ZLqTTgQFw0nBILIEIjNm6g5tzhmbHYHTpHvidg=; b=LMeFsJf0MGqmLUWUFmbhe6/5ZVq1NSp0Fk3n17JnMmHz4MDL1o/kTPslHbt4hnptg/ 6FhupKJ5e39CH9VhAyShpvY5yAuu7KX+VmaRzNFsmFUk47zIdbeFWNPbaFYJV8SpS4G1 vpCHJSWwMgl/5gbWPlXAF5tdU0CRyRv6IHgL4iHZZfuqo9JF48R491AsDm2KXgxy0yzO TQAgFBHsNjW9gSmrfiOyoY4vpCmXFScFNF7dGROB2uqBgpLvIDx1OXQMuR8+N52czhSx IsGMVkv2x835zOWPUAbAc8pyxL9VOo0O1z1ljLPiAcwqTBJEwtHKLIGbAH/5xP1uVPrb VzTw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:13 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:12 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319812" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:12 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 02/26] x86/cpufeatures: Add CET CPU feature flags for Control-flow Enforcement Technology (CET) Date: Mon, 19 Nov 2018 13:47:45 -0800 Message-Id: <20181119214809.6086-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add CPU feature flags for Control-flow Enforcement Technology (CET). CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking Signed-off-by: Yu-cheng Yu Reviewed-by: Borislav Petkov --- arch/x86/include/asm/cpufeatures.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 28c4a502b419..f06f55de7488 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -321,6 +321,7 @@ #define X86_FEATURE_PKU (16*32+ 3) /* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ @@ -343,6 +344,7 @@ #define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */ #define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */ #define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */ +#define X86_FEATURE_IBT (18*32+20) /* Indirect Branch Tracking */ #define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */ #define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */ #define X86_FEATURE_FLUSH_L1D (18*32+28) /* Flush L1D cache */ From patchwork Mon Nov 19 21:47:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689519 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DE3613BB for ; Mon, 19 Nov 2018 21:54:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F27152A583 for ; Mon, 19 Nov 2018 21:54:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E61662A6EC; Mon, 19 Nov 2018 21:54:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9F872A583 for ; Mon, 19 Nov 2018 21:54:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93B976B1C8A; Mon, 19 Nov 2018 16:54:17 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C6A36B1C8C; Mon, 19 Nov 2018 16:54:17 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A36E6B1C8D; Mon, 19 Nov 2018 16:54:17 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 1170E6B1C8A for ; Mon, 19 Nov 2018 16:54:17 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id t2so18715842pfj.15 for ; Mon, 19 Nov 2018 13:54:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=a38K7EgKAhJARBm1PvHH64v2hR075cnrqQp4P7C4yj4=; b=RGyxqIc32Em5nqAb1GBibacjnr9hiRFrlCxtrUaVA33lwsUGRuICDHHNnjvUrnAYp6 SgSXaWVVpy21x+fb98RkA2b9Tee+y/hFWNsrWjiVt/VTIWCH0LklVekqEcSXibmrRQzx HKT2DNYKW+0sIGGxcHpbreAh6By9kd7Gc3zxO07+0y0POzxdQnib8IGPpCkaG0na2ZD/ FkEVWKZfamg9mQPcxHLl3hiUiBSNiXS/m2ktDMhRVJexI67zGbxypIYZ0tUE7P0xeikl yzm/KTDJPqrsldKqwB7bQ1oQUw2+Xu6mhgm33gb+0lz0HPmJuquqEX2WefJRz4NCKyZd JG+g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gIxOQcr9LlRfBgIF3L0yT5SLoFcvwmEt9hda4QisI3NyfiYLzQt vO2CU/Klmc9ySR74aM5eUqZ7fQCBjLVUdS7Qnldk7ABWYT4lfGwOcQfcKFfG0qRy2mb8WM+14U5 MJdfMkLpicMhlI4Z21jvxvxwJRbSJFz0ig7U4sehDmrw3OxWIhdJ+RUqkr7siRVYHLg== X-Received: by 2002:a62:5003:: with SMTP id e3mr25569443pfb.23.1542664456697; Mon, 19 Nov 2018 13:54:16 -0800 (PST) X-Google-Smtp-Source: AJdET5f9rHbQanCUGriaHcZGYgNcnGzlhh14eZG8SLfTe3omx9miZfYQeXnQIptWJ71Y1zOhJKnN X-Received: by 2002:a62:5003:: with SMTP id e3mr25569356pfb.23.1542664455226; Mon, 19 Nov 2018 13:54:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664455; cv=none; d=google.com; s=arc-20160816; b=JzljEbCqp3I9gQspUHZ4/Kd3HYaAuRJYeySZ2roMo4QEbFgBzzgFynTJUNEIg7p3Jt RZdOm2L/V9ejABV6bxqDhhSkxTS97SwsvhEMD53gQx6UNraIjcTzB80P3wpGdGRROhJx NUOp3Koud8dLi6ZxC68v0oPVmRaRpW//RSdrNjpCNU2xzF11TsDeYJoJsdU2Fo/3XBBz LX+iRGvqexZMdSiOAJFZLPfxx3IV/Tc130KdayDAUb7emDXEYmvPRmjrvQ8+u9IrF6Xh TleoaFW+w3nVjBzOGDxMbzz44pxKnb62a/J6k3gBju8x6oPLfMtonXWu7C0CktgStzKO vOug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=a38K7EgKAhJARBm1PvHH64v2hR075cnrqQp4P7C4yj4=; b=p/q1rqGOzoXI7X5bx8/y89n0tcXy+O91L/FdaV2Tb60yEtyroLWIXpzy32+WSzfM3w GJopzeuTZ8ZFeRNWfCH7WUxidpU45qfiO1cxBB/VV4gj7HJPt3DDKlIFmvOZbcQ7i2iH o7Gyhd+FNW+MfdH+Te3nmBtqIxhaZVWcuuXO0+OjB5gyci5jeCg9YtqHp/OXGpPV0J1t hnCA8konAEQfhL283vaTOexBVExO+QB8NCuUsxbphCDhGNnadIrDr4KbtxZNTR7qP2ci cWqCRTxVYx2tjD0jxnNPmgRwziAE/o+lAFnhZC/sxHYbN/FSJ/LOxCseXAL9avXrZrtD c8Qw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:15 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319822" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:12 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 03/26] x86/fpu/xstate: Change names to separate XSAVES system and user states Date: Mon, 19 Nov 2018 13:47:46 -0800 Message-Id: <20181119214809.6086-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control-flow Enforcement (CET) MSR contents are XSAVES system states. To support CET, introduce XSAVES system states first. XSAVES is a "supervisor" instruction and, comparing to XSAVE, saves additional "supervisor" states that can be modified only from CPL 0. However, these states are per-task and not kernel's own. Rename "supervisor" states to "system" states to clearly separate them from "user" states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 4 +- arch/x86/include/asm/fpu/xstate.h | 20 +++---- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +- arch/x86/kernel/fpu/xstate.c | 86 ++++++++++++++--------------- 6 files changed, 60 insertions(+), 62 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 5f7290e6e954..540ef09d4b6e 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -93,7 +93,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) @@ -233,7 +233,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) /* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact - * format and supervisor states in addition to modified optimization in + * format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 48581988d78c..76f83d2ac10e 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -23,15 +23,15 @@ #define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) /* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " @@ -39,7 +39,7 @@ #define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 2ea85b32421a..4bd56079048f 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -363,7 +363,7 @@ void fpu__drop(struct fpu *fpu) * Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs(void) +static inline void copy_init_user_fpstate_to_fpregs(void) { if (use_xsave()) copy_kernel_to_xregs(&init_fpstate.xsave, -1); @@ -395,7 +395,7 @@ void fpu__clear(struct fpu *fpu) preempt_disable(); fpu__initialize(fpu); user_fpu_begin(); - copy_init_fpstate_to_fpregs(); + copy_init_user_fpstate_to_fpregs(); preempt_enable(); } } diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 6abd83572b01..761c3a5a9e07 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -229,7 +229,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) */ u64 __init fpu__get_supported_xfeatures_mask(void) { - return XCNTXT_MASK; + return SUPPORTED_XFEATURES_MASK; } /* Legacy code to initialize eager fpu mode. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 61a949d84dfa..1e20c2272e84 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -249,11 +249,11 @@ static inline int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_ { if (use_xsave()) { if ((unsigned long)buf % 64 || fx_only) { - u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE; + u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE; copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_fxregs(buf); } else { - u64 init_bv = xfeatures_mask & ~xbv; + u64 init_bv = xfeatures_mask_user & ~xbv; if (unlikely(init_bv)) copy_kernel_to_xregs(&init_fpstate.xsave, init_bv); return copy_user_to_xregs(buf, xbv); @@ -416,7 +416,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 = FP_XSTATE_MAGIC1; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) || diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 87a57b7642d3..326b64a61fc9 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -51,13 +51,16 @@ static short xsave_cpuid_features[] __initdata = { }; /* - * Mask of xstate features supported by the CPU and the kernel: + * XSAVES system states can only be modified from CPL 0 and saved by + * XSAVES. The rest are user states. The following is a mask of + * supported user state features derived from boot_cpu_has() and + * SUPPORTED_XFEATURES_MASK. */ -u64 xfeatures_mask __read_mostly; +u64 xfeatures_mask_user __read_mostly; static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -82,7 +85,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -113,15 +116,12 @@ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) } EXPORT_SYMBOL_GPL(cpu_has_xfeatures); -static int xfeature_is_supervisor(int xfeature_nr) +static int xfeature_is_system(int xfeature_nr) { /* - * We currently do not support supervisor states, but if - * we did, we could find out like this. - * * SDM says: If state component 'i' is a user state component, - * ECX[0] return 0; if state component i is a supervisor - * state component, ECX[0] returns 1. + * ECX[0] is 0; if state component i is a system state component, + * ECX[0] is 1. */ u32 eax, ebx, ecx, edx; @@ -131,7 +131,7 @@ static int xfeature_is_supervisor(int xfeature_nr) static int xfeature_is_user(int xfeature_nr) { - return !xfeature_is_supervisor(xfeature_nr); + return !xfeature_is_system(xfeature_nr); } /* @@ -164,7 +164,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask) == xfeatures_mask) + if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) return; /* @@ -191,7 +191,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * in a special way already: */ feature_bit = 0x2; - xfeatures = (xfeatures_mask & ~xfeatures) >> 2; + xfeatures = (xfeatures_mask_user & ~xfeatures) >> 2; /* * Update all the remaining memory layouts according to their @@ -219,20 +219,18 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) return; /* - * Make it clear that XSAVES supervisor states are not yet - * implemented should anyone expect it to work by changing - * bits in XFEATURE_MASK_* macros and XCR0. + * XCR_XFEATURE_ENABLED_MASK sets the features that are managed + * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be + * set here. */ - WARN_ONCE((xfeatures_mask & XFEATURE_MASK_SUPERVISOR), - "x86/fpu: XSAVES supervisor states are not yet implemented.\n"); - xfeatures_mask &= ~XFEATURE_MASK_SUPERVISOR; + xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; cr4_set_bits(X86_CR4_OSXSAVE); - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -242,7 +240,7 @@ void fpu__init_cpu_xstate(void) */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask & (1UL << xfeature)); + return !!(xfeatures_mask_user & BIT_ULL(xfeature)); } /* @@ -272,7 +270,7 @@ static void __init setup_xstate_features(void) cpuid_count(XSTATE_CPUID, i, &eax, &ebx, &ecx, &edx); /* - * If an xfeature is supervisor state, the offset + * If an xfeature is a system state, the offset * in EBX is invalid. We leave it to -1. */ if (xfeature_is_user(i)) @@ -348,7 +346,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; int i; /* @@ -421,7 +419,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = (u64)1 << 63 | xfeatures_mask; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; /* * Init all the features state with header.xfeatures being 0x0 @@ -440,8 +438,8 @@ static int xfeature_uncompacted_offset(int xfeature_nr) u32 eax, ebx, ecx, edx; /* - * Only XSAVES supports supervisor states and it uses compacted - * format. Checking a supervisor state's uncompacted offset is + * Only XSAVES supports system states and it uses compacted + * format. Checking a system state's uncompacted offset is * an error. */ if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { @@ -465,7 +463,7 @@ static int xfeature_size(int xfeature_nr) /* * 'XSAVES' implies two different things: - * 1. saving of supervisor/system state + * 1. saving of system state * 2. using the compacted format * * Use this function when dealing with the compacted format so @@ -480,8 +478,8 @@ int using_compacted_format(void) /* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ int validate_xstate_header(const struct xstate_header *hdr) { - /* No unknown or supervisor features may be set */ - if (hdr->xfeatures & (~xfeatures_mask | XFEATURE_MASK_SUPERVISOR)) + /* No unknown or system features may be set */ + if (hdr->xfeatures & ~xfeatures_mask_user) return -EINVAL; /* Userspace must use the uncompacted format */ @@ -588,11 +586,11 @@ static void do_extra_xstate_size_checks(void) check_xstate_against_struct(i); /* - * Supervisor state components can be managed only by + * System state components can be managed only by * XSAVES, which is compacted-format only. */ if (!using_compacted_format()) - XSTATE_WARN_ON(xfeature_is_supervisor(i)); + XSTATE_WARN_ON(xfeature_is_system(i)); /* Align from the end of the previous feature */ if (xfeature_is_aligned(i)) @@ -616,7 +614,7 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask. + * Get total size of enabled xstates in XCR0/xfeatures_mask_user. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer @@ -706,7 +704,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { - xfeatures_mask = 0; + xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); } @@ -742,15 +740,15 @@ void __init fpu__init_system_xstate(void) } cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask = eax + ((u64)edx << 32); + xfeatures_mask_user = eax + ((u64)edx << 32); - if ((xfeatures_mask & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue * booting without it. This is too early to BUG(). */ - pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask); + pr_err("x86/fpu: FP/SSE not present amongst the CPU's xstate features: 0x%llx.\n", xfeatures_mask_user); goto out_disable; } @@ -759,10 +757,10 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask &= ~BIT(i); + xfeatures_mask_user &= ~BIT_ULL(i); } - xfeatures_mask &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -772,9 +770,9 @@ void __init fpu__init_system_xstate(void) /* * Update info used for ptrace frames; use standard-format size and no - * supervisor xstates: + * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -782,7 +780,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask, + xfeatures_mask_user, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -801,7 +799,7 @@ void fpu__resume_cpu(void) * Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has(X86_FEATURE_XSAVE)) - xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask); + xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); } /* @@ -853,7 +851,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) * have not enabled. Remember that pcntxt_mask is * what we write to the XCR0 register. */ - WARN_ONCE(!(xfeatures_mask & xstate_feature), + WARN_ONCE(!(xfeatures_mask_user & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to From patchwork Mon Nov 19 21:47:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689521 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEEB91923 for ; Mon, 19 Nov 2018 21:54:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B085A2A583 for ; Mon, 19 Nov 2018 21:54:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A48532A6EC; Mon, 19 Nov 2018 21:54:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A19432A583 for ; Mon, 19 Nov 2018 21:54:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E28E6B1C8C; Mon, 19 Nov 2018 16:54:18 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 51C046B1C8D; Mon, 19 Nov 2018 16:54:18 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39A6A6B1C8E; Mon, 19 Nov 2018 16:54:18 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id DC6CA6B1C8C for ; Mon, 19 Nov 2018 16:54:17 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id a18so21584104pga.16 for ; Mon, 19 Nov 2018 13:54:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=DvIyi/lb+BFXy8f/Hz7e4WNoqzV8VUl53ZGhlUxcnu8=; b=BLgL6WHPA3vdRMtjew9b7ate/bwGGOKAkw35fKp6ffaWmWPeg2B5bAnKivLITIussQ dhd5+jAWQgX4vD9bAuNNtAAGcRzn+r3ibruMwjeFBE++Ly016CHLCeyxUiOw9zk0UvUq m3Ol2RdyEHK9RVabFfgp9sKdhQjCD3bAdI2qdtiBU9BWUivEynleSfGZywKLVB/FVPJk 9CZAbi8J4iYdYxupo95zhbQNxLxLjG7+tfglAV//iVKk83SjY57WH2mRBheWNFA/UTlX vOykagMxA19trE61/IJe73pDsZjpah5oUTZhnQ0lXocrAVc/3xXqDmg3SiTdYXEexrrq OuCA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWaObD0sk7I//phevHCK15Orw+ikhoVNyczI32tSBoDEiNWLF39y 0MkW+vjai7SZxV7r3lcA2tDj034PtE84i543q1xIOIUFnZgp6N90VuQJ2Tx9m7OBwuS68kAJD80 6YIAW0Xn8JtstaIXvHg3ErUbl88pqaZTfFfmgxAFL3KYD/pvyGidPU4rjugCJIvMZ+g== X-Received: by 2002:a65:564b:: with SMTP id m11mr12291940pgs.216.1542664457501; Mon, 19 Nov 2018 13:54:17 -0800 (PST) X-Google-Smtp-Source: AFSGD/U0VoIyrkGnJUUPwo7xrtNGBarbSkQVu6EFM31srIAFMbq2nzw7wFVDTlpkf0hSDimZ8548 X-Received: by 2002:a65:564b:: with SMTP id m11mr12291883pgs.216.1542664456133; Mon, 19 Nov 2018 13:54:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664456; cv=none; d=google.com; s=arc-20160816; b=jTk6zTeYYtmeGXE1oV8lEgbb7xO+a8YfsLmW+z0FJDweQxs5BEwoaejXwWzybpcv00 mnx5ClmstrKuYHgEsgXGjO6MwwqW+xeizqr3rUM1TpXXV3pF0N6e5KcQ0uzEdNT9QKX7 yOaoaj/Wq6egdYwoOcgB5d/jVSH8PB+OvbrEm73UGcTBW+3V9xbPbURw1JpgUvLvhO94 WkUMvMHAP2GhllkJ1V3iW4OSnkVimsBAPdhnQESfEUdNj1KnV9QmDnRUSQoq83LJH3EI GnMLCpXmjZnjFXK8sXstV4wEHZhCRCd5IjTG5TRicbfGNAezKGshhUT4pwPqZX+lpGsa 14WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=DvIyi/lb+BFXy8f/Hz7e4WNoqzV8VUl53ZGhlUxcnu8=; b=lQ3uj+kBCtF2QzjFEqfICDB0Md4ypi3dyG44fihtPMc+nWPsdXxsDDri+1AZqTe/4d hdmeET8UW0Rmb1yRhoBjf/c2gxtbAX57Th72ljCsB92sn4/NH6nMOrJGkxeCCnfMZtdB ImWPZu2GtOZuQCZ8pRXTp8ki5g/FNo1ndn36z6wfKhakeCUtF4h+8CGTrdACBL86Qm+y tEVjwyKcaYpCu6H5EmSH6jSXSoO6EwWn3MtokhC7HeefaDnnAbF81PKyr4uVUZODXUhj 4NLCqwxpglwJUKUu0jwn/CTxbkeCu1AYNTQ/fGgVkLkQLCUg8IlecJ/gMUcvULarwn4L h1sg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:16 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319830" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:14 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 04/26] x86/fpu/xstate: Introduce XSAVES system states Date: Mon, 19 Nov 2018 13:47:47 -0800 Message-Id: <20181119214809.6086-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Control-flow Enforcement (CET) MSR contents are XSAVES system states. To support CET, introduce XSAVES system states first. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/internal.h | 3 +- arch/x86/include/asm/fpu/xstate.h | 4 +- arch/x86/kernel/fpu/core.c | 6 +- arch/x86/kernel/fpu/init.c | 10 --- arch/x86/kernel/fpu/xstate.c | 94 +++++++++++++++++++---------- 5 files changed, 69 insertions(+), 48 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 540ef09d4b6e..3f38c9aec553 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -45,7 +45,6 @@ extern void fpu__init_cpu_xstate(void); extern void fpu__init_system(struct cpuinfo_x86 *c); extern void fpu__init_check_bugs(void); extern void fpu__resume_cpu(void); -extern u64 fpu__get_supported_xfeatures_mask(void); /* * Debugging facility: @@ -93,7 +92,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave) * XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_user; + xsave->header.xcomp_bv = XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; } static inline void fpstate_init_fxstate(struct fxregs_state *fx) diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 76f83d2ac10e..d8e2ec99f635 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -19,9 +19,6 @@ #define XSAVE_YMM_SIZE 256 #define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) - /* All currently supported features */ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ @@ -40,6 +37,7 @@ #endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all; extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS]; extern void __init update_regset_xstate_info(unsigned int size, diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 4bd56079048f..5f076f593fa9 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -365,8 +365,12 @@ void fpu__drop(struct fpu *fpu) */ static inline void copy_init_user_fpstate_to_fpregs(void) { + /* + * Only XSAVES user states are copied. + * System states are preserved. + */ if (use_xsave()) - copy_kernel_to_xregs(&init_fpstate.xsave, -1); + copy_kernel_to_xregs(&init_fpstate.xsave, xfeatures_mask_user); else if (static_cpu_has(X86_FEATURE_FXSR)) copy_kernel_to_fxregs(&init_fpstate.fxsave); else diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c index 761c3a5a9e07..eaf9d9d479a5 100644 --- a/arch/x86/kernel/fpu/init.c +++ b/arch/x86/kernel/fpu/init.c @@ -222,16 +222,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* - * Find supported xfeatures based on cpu features and command-line input. - * This must be called after fpu__init_parse_early_param() is called and - * xfeatures_mask is enumerated. - */ -u64 __init fpu__get_supported_xfeatures_mask(void) -{ - return SUPPORTED_XFEATURES_MASK; -} - /* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch(void) { diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 326b64a61fc9..f6d2e2e53463 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -58,9 +58,19 @@ static short xsave_cpuid_features[] __initdata = { */ u64 xfeatures_mask_user __read_mostly; +/* + * Supported XSAVES system states. + */ +static u64 xfeatures_mask_system __read_mostly; + +/* + * Combined XSAVES system and user states. + */ +u64 xfeatures_mask_all __read_mostly; + static unsigned int xstate_offsets[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; static unsigned int xstate_sizes[XFEATURE_MAX] = { [ 0 ... XFEATURE_MAX - 1] = -1}; -static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_user)*8]; +static unsigned int xstate_comp_offsets[sizeof(xfeatures_mask_all)*8]; /* * The XSAVE area of kernel can be in standard or compacted format; @@ -85,7 +95,7 @@ void fpu__xstate_clear_all_cpu_caps(void) */ int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) { - u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_user; + u64 xfeatures_missing = xfeatures_needed & ~xfeatures_mask_all; if (unlikely(feature_name)) { long xfeature_idx, max_idx; @@ -164,7 +174,7 @@ void fpstate_sanitize_xstate(struct fpu *fpu) * None of the feature bits are in init state. So nothing else * to do for us, as the memory layout is up to date. */ - if ((xfeatures & xfeatures_mask_user) == xfeatures_mask_user) + if ((xfeatures & xfeatures_mask_all) == xfeatures_mask_all) return; /* @@ -219,28 +229,27 @@ void fpstate_sanitize_xstate(struct fpu *fpu) */ void fpu__init_cpu_xstate(void) { - if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_user) + if (!boot_cpu_has(X86_FEATURE_XSAVE) || !xfeatures_mask_all) return; /* * XCR_XFEATURE_ENABLED_MASK sets the features that are managed * by XSAVE{C, OPT} and XRSTOR. Only XSAVE user states can be * set here. */ - - xfeatures_mask_user &= ~XFEATURE_MASK_SUPERVISOR; - cr4_set_bits(X86_CR4_OSXSAVE); xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * MSR_IA32_XSS controls which system (not user) states are + * to be managed by XSAVES. + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_system); } -/* - * Note that in the future we will likely need a pair of - * functions here: one for user xstates and the other for - * system xstates. For now, they are the same. - */ static int xfeature_enabled(enum xfeature xfeature) { - return !!(xfeatures_mask_user & BIT_ULL(xfeature)); + return !!(xfeatures_mask_all & BIT_ULL(xfeature)); } /* @@ -346,7 +355,7 @@ static int xfeature_is_aligned(int xfeature_nr) */ static void __init setup_xstate_comp(void) { - unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_user)*8]; + unsigned int xstate_comp_sizes[sizeof(xfeatures_mask_all)*8]; int i; /* @@ -419,7 +428,7 @@ static void __init setup_init_fpu_buf(void) print_xstate_features(); if (boot_cpu_has(X86_FEATURE_XSAVES)) - init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_user; + init_fpstate.xsave.header.xcomp_bv = BIT_ULL(63) | xfeatures_mask_all; /* * Init all the features state with header.xfeatures being 0x0 @@ -442,7 +451,7 @@ static int xfeature_uncompacted_offset(int xfeature_nr) * format. Checking a system state's uncompacted offset is * an error. */ - if (XFEATURE_MASK_SUPERVISOR & (1 << xfeature_nr)) { + if (~xfeatures_mask_user & BIT_ULL(xfeature_nr)) { WARN_ONCE(1, "No fixed offset for xstate %d\n", xfeature_nr); return -1; } @@ -614,15 +623,12 @@ static void do_extra_xstate_size_checks(void) /* - * Get total size of enabled xstates in XCR0/xfeatures_mask_user. + * Get total size of enabled xstates in XCR0 | IA32_XSS. * * Note the SDM's wording here. "sub-function 0" only enumerates * the size of the *user* states. If we use it to size a buffer * that we use 'XSAVES' on, we could potentially overflow the * buffer because 'XSAVES' saves system states too. - * - * Note that we do not currently set any bits on IA32_XSS so - * 'XCR0 | IA32_XSS == XCR0' for now. */ static unsigned int __init get_xsaves_size(void) { @@ -704,6 +710,7 @@ static int init_xstate_size(void) */ static void fpu__init_disable_system_xstate(void) { + xfeatures_mask_all = 0; xfeatures_mask_user = 0; cr4_clear_bits(X86_CR4_OSXSAVE); fpu__xstate_clear_all_cpu_caps(); @@ -717,6 +724,8 @@ void __init fpu__init_system_xstate(void) { unsigned int eax, ebx, ecx, edx; static int on_boot_cpu __initdata = 1; + u64 cpu_system_xfeatures_mask; + u64 cpu_user_xfeatures_mask; int err; int i; @@ -739,10 +748,23 @@ void __init fpu__init_system_xstate(void) return; } + /* + * Find user states supported by the processor. + * Only these bits can be set in XCR0. + */ cpuid_count(XSTATE_CPUID, 0, &eax, &ebx, &ecx, &edx); - xfeatures_mask_user = eax + ((u64)edx << 32); + cpu_user_xfeatures_mask = eax + ((u64)edx << 32); - if ((xfeatures_mask_user & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { + /* + * Find system states supported by the processor. + * Only these bits can be set in IA32_XSS MSR. + */ + cpuid_count(XSTATE_CPUID, 1, &eax, &ebx, &ecx, &edx); + cpu_system_xfeatures_mask = ecx + ((u64)edx << 32); + + xfeatures_mask_all = cpu_user_xfeatures_mask | cpu_system_xfeatures_mask; + + if ((xfeatures_mask_all & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -757,10 +779,12 @@ void __init fpu__init_system_xstate(void) */ for (i = 0; i < ARRAY_SIZE(xsave_cpuid_features); i++) { if (!boot_cpu_has(xsave_cpuid_features[i])) - xfeatures_mask_user &= ~BIT_ULL(i); + xfeatures_mask_all &= ~BIT_ULL(i); } - xfeatures_mask_user &= fpu__get_supported_xfeatures_mask(); + xfeatures_mask_all &= SUPPORTED_XFEATURES_MASK; + xfeatures_mask_user = xfeatures_mask_all & cpu_user_xfeatures_mask; + xfeatures_mask_system = xfeatures_mask_all & cpu_system_xfeatures_mask; /* Enable xstate instructions to be able to continue with initialization: */ fpu__init_cpu_xstate(); @@ -772,7 +796,7 @@ void __init fpu__init_system_xstate(void) * Update info used for ptrace frames; use standard-format size and no * system xstates: */ - update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user & ~XFEATURE_MASK_SUPERVISOR); + update_regset_xstate_info(fpu_user_xstate_size, xfeatures_mask_user); fpu__init_prepare_fx_sw_frame(); setup_init_fpu_buf(); @@ -780,7 +804,7 @@ void __init fpu__init_system_xstate(void) print_xstate_offset_size(); pr_info("x86/fpu: Enabled xstate features 0x%llx, context size is %d bytes, using '%s' format.\n", - xfeatures_mask_user, + xfeatures_mask_all, fpu_kernel_xstate_size, boot_cpu_has(X86_FEATURE_XSAVES) ? "compacted" : "standard"); return; @@ -800,6 +824,12 @@ void fpu__resume_cpu(void) */ if (boot_cpu_has(X86_FEATURE_XSAVE)) xsetbv(XCR_XFEATURE_ENABLED_MASK, xfeatures_mask_user); + + /* + * Restore IA32_XSS + */ + if (boot_cpu_has(X86_FEATURE_XSAVES)) + wrmsrl(MSR_IA32_XSS, xfeatures_mask_system); } /* @@ -849,9 +879,9 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) /* * We should not ever be requesting features that we * have not enabled. Remember that pcntxt_mask is - * what we write to the XCR0 register. + * what we write to the XCR0 | IA32_XSS registers. */ - WARN_ONCE(!(xfeatures_mask_user & xstate_feature), + WARN_ONCE(!(xfeatures_mask_all & xstate_feature), "get of unsupported state"); /* * This assumes the last 'xsave*' instruction to @@ -1001,7 +1031,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1085,7 +1115,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i */ memset(&header, 0, sizeof(header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &= ~XFEATURE_MASK_SUPERVISOR; + header.xfeatures &= xfeatures_mask_user; /* * Copy xregs_state->header: @@ -1178,7 +1208,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: @@ -1234,7 +1264,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) * The state that came in from userspace was user-state only. * Mask all the user states out of 'xfeatures': */ - xsave->header.xfeatures &= XFEATURE_MASK_SUPERVISOR; + xsave->header.xfeatures &= (xfeatures_mask_all & ~xfeatures_mask_user); /* * Add back in the features that came in from userspace: From patchwork Mon Nov 19 21:47:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689523 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CB3A14BD for ; Mon, 19 Nov 2018 21:54:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D4BC2A583 for ; Mon, 19 Nov 2018 21:54:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4103C2A6EC; Mon, 19 Nov 2018 21:54:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A49B92A583 for ; Mon, 19 Nov 2018 21:54:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 792836B1C8E; Mon, 19 Nov 2018 16:54:19 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 717546B1C90; Mon, 19 Nov 2018 16:54:19 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 544AF6B1C8F; Mon, 19 Nov 2018 16:54:19 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 020BE6B1C8D for ; Mon, 19 Nov 2018 16:54:19 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id w7-v6so24592794plp.9 for ; Mon, 19 Nov 2018 13:54:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=VnNoh0RScPsSRVgPKI7h8XelVsKRiRxUuAuUrl/fLHk=; b=mS4TvqWwtjtE/cWyPAq4b4/u4YTZ1ERx8LeNg2oD4FTUpTXp2bhLYTv3DWS2eKUTmm P94ZkJkG57aZ7/AV4v5dMaZsmWPlHRt0WK8fG7NNQq6WUNlug3C32iQiAAWWIRQMhwho UWIHWddrPom7yLnaabozTO9d5Jmhd7fKTXf1bKdP1zqLI92q4b5QurEIAwG7ZOAXQ/3M yAfB6IvTxnfv/wGzJW30+yBoK3XFH41DfAdecfwGtUh5PAdcdy2erZ9WIiIXWZY2QraE TBocIJs8QDyq9JNq4V3/YFHyGfgfFc9rbKfRZrXDIaDCjsQk1ZzKMGcLqDYBXPi9Zb/q ruRA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJ5oHS4N6QWksHfK3YBnGKXxmaE/Klp6fh8QYbWkE3aYHJbk5N5 m6kIZ5I8cpCPLWciO8teuan9JaHkrCVZd/UZ9SLulzpkYovr3GB3mXqrG3WdV8lUwsXSlNx3IZU 1vdc7v8cFHJfU9XyOF9uZ61otheQVmJG4p4U3lF6sDyYbXcnlmXfCOyvrjqM+olpdzA== X-Received: by 2002:a63:981:: with SMTP id 123mr21769252pgj.444.1542664458651; Mon, 19 Nov 2018 13:54:18 -0800 (PST) X-Google-Smtp-Source: AJdET5dOhjI6cLfPSguoi7Hgg4IKFuvZLPyR6TXM1QZGe8z6x3dHbbeoojfaMhjm9QT8Gp6tqCt/ X-Received: by 2002:a63:981:: with SMTP id 123mr21769194pgj.444.1542664457269; Mon, 19 Nov 2018 13:54:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664457; cv=none; d=google.com; s=arc-20160816; b=GBj1jFFwvghgH+joH9m1h0eMwNvJiBrv+O8ViCa57fIu04fpUcDLdBKBV5KMfLqT0d DM8rZdkknglU3YcUJsRG+Hc7iH1/eHyW+SQoSJPqHpJMG2S43WUPlU/UqFmf1RPQESgL W7ZFc8xVdMgSn6lKACZ/BfhC5tsA14f6NvQhMQSVy2YmQLSqv2hYHjM0QsqrUDN2qdfU YW2u9zIqqTgJ8SYPXhw79YrXmwtPBGu65Tv+GX2q55Gg8l4msv1b8ZKHL5V087foTmsM IU4Uy45EGkyVDrjS2a2d4m4XuJ+GRsUD8tR4eMkY8ECTLFbq9cxrEmNKaQ0Z0vBdfYvc t9Sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=VnNoh0RScPsSRVgPKI7h8XelVsKRiRxUuAuUrl/fLHk=; b=GQT8NlAY4R874xUrBCYdnAK4E5BcLLaBeTWv7m0GK3mnaHZWRcZQIc44RaYM4WcqD/ AxM8zJNCG8clndQntgyWz1S1MVEuUmw3kpZNogus4Tp/xLOOwGFPaaGqmT+fWANuZFgp rZww9MDXBH4j7fNBU8Qn5K8Zg/sApGBnzB6HpzJ29U05JJenQoA19Der6ESaRo9z+3EQ uD59+mry+gRzMBpCevtYLyZCb2+AdUm9SwZf1b425oXjEkxgCUiuwrh4w4gVqONV1Ul0 wvrlV+89fkWGI46vv8Dzwyxvb4iO3v0bL+DFLfvvn1qymprTWMQK+4WwJbN4HxfHoDdQ p2eg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:17 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319837" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:15 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 05/26] x86/fpu/xstate: Add XSAVES system states for shadow stack Date: Mon, 19 Nov 2018 13:47:48 -0800 Message-Id: <20181119214809.6086-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Intel Control-flow Enforcement Technology (CET) introduces the following MSRs. MSR_IA32_U_CET (user-mode CET settings), MSR_IA32_PL3_SSP (user-mode shadow stack), MSR_IA32_PL0_SSP (kernel-mode shadow stack), MSR_IA32_PL1_SSP (Privilege Level 1 shadow stack), MSR_IA32_PL2_SSP (Privilege Level 2 shadow stack). Introduce them into XSAVES system states. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index 202c53918ecf..792e4ea466ce 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM, XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, + XFEATURE_RESERVED, + XFEATURE_SHSTK_USER, + XFEATURE_SHSTK_KERNEL, XFEATURE_MAX, }; @@ -128,6 +131,8 @@ enum xfeature { #define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM) #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_SHSTK_USER (1 << XFEATURE_SHSTK_USER) +#define XFEATURE_MASK_SHSTK_KERNEL (1 << XFEATURE_SHSTK_KERNEL) #define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE) #define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \ @@ -229,6 +234,23 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control-flow Enforcement user states + */ +struct cet_user_state { + u64 user_cet; /* user control-flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +}; + +/* + * State component 12 is Control-flow Enforcement kernel states + */ +struct cet_kernel_state { + u64 kernel_ssp; /* kernel shadow stack */ + u64 pl1_ssp; /* privilege level 1 shadow stack */ + u64 pl2_ssp; /* privilege level 2 shadow stack */ +}; + struct xstate_header { u64 xfeatures; u64 xcomp_bv; diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index d8e2ec99f635..18b60748a34d 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -28,7 +28,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_SHSTK_USER | \ + XFEATURE_MASK_SHSTK_KERNEL) #ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, " diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index bcba3c643e63..a8df907e8017 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -130,6 +130,8 @@ #define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT) #define X86_CR4_PKE_BIT 22 /* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) /* * x86-64 Task Priority Register, CR8 diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index f6d2e2e53463..8c2dff13eac6 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -35,6 +35,9 @@ static const char *xfeature_names[] = "Processor Trace (unused)" , "Protection Keys User registers", "unknown xstate feature" , + "Control-flow User registers" , + "Control-flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = { @@ -48,6 +51,9 @@ static short xsave_cpuid_features[] __initdata = { X86_FEATURE_AVX512F, X86_FEATURE_INTEL_PT, X86_FEATURE_PKU, + 0, /* Unused */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_KERNEL */ }; /* @@ -319,6 +325,8 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_ZMM_Hi256); print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); + print_xstate_feature(XFEATURE_MASK_SHSTK_USER); + print_xstate_feature(XFEATURE_MASK_SHSTK_KERNEL); } /* @@ -565,6 +573,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_KERNEL, struct cet_kernel_state); /* * Make *SURE* to add any feature numbers in below if From patchwork Mon Nov 19 21:47:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689525 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48D6C13BB for ; Mon, 19 Nov 2018 21:54:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3965D2A583 for ; Mon, 19 Nov 2018 21:54:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D3762A6EC; Mon, 19 Nov 2018 21:54:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FC282A583 for ; Mon, 19 Nov 2018 21:54:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC0276B1C8D; Mon, 19 Nov 2018 16:54:19 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E1C206B1C90; Mon, 19 Nov 2018 16:54:19 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAE576B1C91; Mon, 19 Nov 2018 16:54:19 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 66AF16B1C8D for ; Mon, 19 Nov 2018 16:54:19 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id q62so20210839pgq.9 for ; Mon, 19 Nov 2018 13:54:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=8oSk42kh3BYlf3vkfF86/Vmc47r2swRGIMDB+iKDLT4=; b=STedHpbvLdujLsrzCEGNP4MBLhIxwSp8jWRGCbRRLAU7leH/UqlFykDOtPu+rHk/Gx EPO9qYBR8Ki9tJ0o56ShDhixEn+oDIyG5218JH2ORG0GKLUW40KzRSW1ufVZds49rCYy ljbC4OAbgkp6obatO1Zv0KdLNHi/2SFJY9JigewX1WUlTA4CV3WD0jrM2ethFkVwNwRx WtHMmFnZvf8XVoiw7kPaVpANhcn2W/upD8FwD25EbZ2G43RZ4e9P5Uk3CxLrmdiOyWUe KFOZS4+AYkINfSgz88LrEUfjAvhj3vBu7xgJmnbgtzjgAFQ8QvPfYNjxegDCn0UEfpeu lSiA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJTA3GrUSyIr34FEuaM3yNx1+bcsKbfIjEQwwTd5dMt7ZoE7Jsx IuMHrsjwvD7Te7BTpuwFqA8aHuFl6sxdCGyELOqvVPnt3syUzXK89eP0L4JuXDp5fiWbA7OrO1r pbhPIoN9QJ1B9CMbF+2PK9pZfolGoA/1Qpivw+C1MhNIAWOb8ba/hzzVaItjvvpA2oQ== X-Received: by 2002:a63:151f:: with SMTP id v31mr21357691pgl.34.1542664459038; Mon, 19 Nov 2018 13:54:19 -0800 (PST) X-Google-Smtp-Source: AJdET5c+0nQbwXxDqGqMwxeyaGIf+5RPeort5rma9nj4XvnBh7owuVGIJDRL/BsuYBERPE3g3Txr X-Received: by 2002:a63:151f:: with SMTP id v31mr21357643pgl.34.1542664458140; Mon, 19 Nov 2018 13:54:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664458; cv=none; d=google.com; s=arc-20160816; b=tKweoPyVIMQyQ5F8nukbuto8YpOCXT+NywFQ7G7mLndu95LvvzsIS/Duc4dFr7mTG7 dKWjqhqvSY1yBbo0VLnZJSG3UbcykdlqesPpNqeHnTEAc2dJ0x6zWYcCJafNXqJWbw04 Q0wK1KgOmcldmhEgf8//15H/CpJTWQW9uPKzZ4wcUnlOM3mGhYvIsAh/GyDb2s7ZuwVr jcDFYtfHhhf4E77qnPgw3ZuQMKFtzsvfohfALqlx70XMUdOnNRooBW8ptmqYTaQLpGMO I4vUo7/+o8Iw0dsmxgHJFHyuruDbFazUw3N6hVuFat24/2UwxG5PTwt3hMA5FofJT8+s +ScQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=8oSk42kh3BYlf3vkfF86/Vmc47r2swRGIMDB+iKDLT4=; b=YCetvOwh+iHT8YnlkHWbFU0DTGnLxahmFTiBljMgWQ5FJ+aGjWiSKdsrpk+YCpGMF/ w6PKqk9H6j3raDWUmV/FxmcRaDEmAkSETb1Ofv14liHYAeYALQLkZWsmBZ5hMGF+s+Tm pXVXqq6VPh7dUn+TPGgLNv5nB1dIH339yf5ZmVIWcSZExaemI5c2kRXhvnUCBrsMMjpM N41S60nRHC8luHRnIkfpMZ1wUGZEV3BAQm/HK2n/DtJHSz0y11UMKFRHZWNCvKUcF79r lcQSz1H4napV5W2O7Gm4O0E2l4kjmuA9Mh4gEIR+7b1lLEbZNW1cbOQeazu5JQaw5/9r jV8A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:18 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:17 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319843" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:16 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 06/26] x86/cet: Add control protection exception handler Date: Mon, 19 Nov 2018 13:47:49 -0800 Message-Id: <20181119214809.6086-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction. The control protection exception handler works in a similar way as the general protection fault handler. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/signal_compat.c | 2 +- arch/x86/kernel/traps.c | 57 ++++++++++++++++++++++++++++++ include/uapi/asm-generic/siginfo.h | 3 +- 6 files changed, 68 insertions(+), 3 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index ce25d84023c0..660b56706d91 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -976,7 +976,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 3de69330e6c5..20841d301e93 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void); @@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long); #endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *regs, long error_code); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long); @@ -142,6 +144,7 @@ enum { X86_TRAP_AC, /* 17, Alignment Check */ X86_TRAP_MC, /* 18, Machine Check */ X86_TRAP_XF, /* 19, SIMD Floating-Point Exception */ + X86_TRAP_CP = 21, /* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ }; diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c index 01adea278a71..66ebc8cb16e2 100644 --- a/arch/x86/kernel/idt.c +++ b/arch/x86/kernel/idt.c @@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = { #elif defined(CONFIG_X86_32) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32), #endif + +#ifdef CONFIG_X86_64 + INTG(X86_TRAP_CP, control_protection), +#endif }; /* diff --git a/arch/x86/kernel/signal_compat.c b/arch/x86/kernel/signal_compat.c index 9ccbf0576cd0..c572a3de1037 100644 --- a/arch/x86/kernel/signal_compat.c +++ b/arch/x86/kernel/signal_compat.c @@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void) */ BUILD_BUG_ON(NSIGILL != 11); BUILD_BUG_ON(NSIGFPE != 15); - BUILD_BUG_ON(NSIGSEGV != 7); + BUILD_BUG_ON(NSIGSEGV != 8); BUILD_BUG_ON(NSIGBUS != 5); BUILD_BUG_ON(NSIGTRAP != 5); BUILD_BUG_ON(NSIGCHLD != 6); diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 9b7c4ca8f0a7..0bef2e914d5f 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -569,6 +569,63 @@ do_general_protection(struct pt_regs *regs, long error_code) } NOKPROBE_SYMBOL(do_general_protection); +static const char *control_protection_err[] = { + "unknown", + "near-ret", + "far-ret/iret", + "endbranch", + "rstorssp", + "setssbsy", +}; + +/* + * When a control protection exception occurs, send a signal + * to the responsible application. Currently, control + * protection is only enabled for the user mode. This + * exception should not come from the kernel mode. + */ +dotraplinkage void +do_control_protection(struct pt_regs *regs, long error_code) +{ + struct task_struct *tsk; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn't wake RCU"); + if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code, X86_TRAP_CP, SIGSEGV) == NOTIFY_STOP) + return; + cond_local_irq_enable(regs); + + if (!user_mode(regs)) + die("kernel control protection fault", regs, error_code); + + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) && + printk_ratelimit()) { + unsigned int max_err; + + max_err = ARRAY_SIZE(control_protection_err) - 1; + if ((error_code < 0) || (error_code > max_err)) + error_code = 0; + pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)", + tsk->comm, task_pid_nr(tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); + print_vma_addr(KERN_CONT " in ", regs->ip); + pr_cont("\n"); + } + + force_sig_fault(SIGSEGV, SEGV_CPERR, + (void __user *)uprobe_get_trap_addr(regs), tsk); +} +NOKPROBE_SYMBOL(do_control_protection); + dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) { #ifdef CONFIG_DYNAMIC_FTRACE diff --git a/include/uapi/asm-generic/siginfo.h b/include/uapi/asm-generic/siginfo.h index cb3d6c267181..693071dbe641 100644 --- a/include/uapi/asm-generic/siginfo.h +++ b/include/uapi/asm-generic/siginfo.h @@ -229,7 +229,8 @@ typedef struct siginfo { #define SEGV_ACCADI 5 /* ADI not enabled for mapped object */ #define SEGV_ADIDERR 6 /* Disrupting MCD error */ #define SEGV_ADIPERR 7 /* Precise MCD exception */ -#define NSIGSEGV 7 +#define SEGV_CPERR 8 +#define NSIGSEGV 8 /* * SIGBUS si_codes From patchwork Mon Nov 19 21:47:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689527 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD2C714BD for ; Mon, 19 Nov 2018 21:54:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C15302A583 for ; Mon, 19 Nov 2018 21:54:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B58312A6EC; Mon, 19 Nov 2018 21:54:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 451702A583 for ; Mon, 19 Nov 2018 21:54:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 968C16B1C90; Mon, 19 Nov 2018 16:54:20 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C3436B1C91; Mon, 19 Nov 2018 16:54:20 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7422E6B1C92; Mon, 19 Nov 2018 16:54:20 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 248DF6B1C90 for ; Mon, 19 Nov 2018 16:54:20 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id g63-v6so27543281pfc.9 for ; Mon, 19 Nov 2018 13:54:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Kue5H9yCIZODI4T4AifLYhnGwkJ2G/QsVSJ2CiNfkS8=; b=KzgStzho1FihS1yyHSygQai/7B8F0TIbC+nUExLtQF2mxnHSbH1osQWDlLsmx2MG24 a18dOXZn3I2uHDXTd1YiRd8hbMCNS3ENqLmOQU4A8dSU3sec6339d192E5OnfbBtS455 iffpFgGZKVCfk30hy+Fy9cXsR9PvT/FxJbLcnxTX8rgx6oDHMUXjgfnEXVKo1k/ofWg4 ot5FZd4wfwEMqboD0vfgdCKbtmfcEOxpSvyE2lZJYCBA3v2UgTCT+qnfYwEvR0aHcwnf uBLa2lOmZrMB9CkJNPxEIp12/ObMhXURINlgLhgquQRvGG8SAVaCC/lEMhjZglPrISri d4HQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gKNcCiRRLp7raAII3mYlnraRRYsGunCC2mhC9BPwO0lkCrCgBPg CeR3B3iJ/zTss2srrTr4rjQ8CiK7nrP+p1PAJofk2H2YR0csziR/c9bDRDr+PROGVFok5VGb/DX wc5gxfca2uIoQ2pn7DGp8sa578Z4PjNLfac/6LDOz7+a5/J4Anp9snZRw2hYPKPrPXA== X-Received: by 2002:a63:2315:: with SMTP id j21mr21864958pgj.297.1542664459815; Mon, 19 Nov 2018 13:54:19 -0800 (PST) X-Google-Smtp-Source: AJdET5dGLKHfnMlQ+TckKfPmmcErz67Fs3YMgZpLmaGkiVjTLWwIAcokZInbmqAGCEUACyD8QdiS X-Received: by 2002:a63:2315:: with SMTP id j21mr21864922pgj.297.1542664458989; Mon, 19 Nov 2018 13:54:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664458; cv=none; d=google.com; s=arc-20160816; b=WPRQI1bbj6zZ7I93mveUbLFIQz14mS4Z365nmqahEhUaWLsW2CQCcQ4RzEEutrpnkU nZh+/kIiIJRWL4q5JBcAlyd7O6UtNAlgX6snAVPzqDXCOEvdot8b5dLWTOc4wgVFdF1Q Rd835hNkkaMHamuGyH9Ct/zSZmIy6SKYgSq/filNYF2BfqNshMSrIuhf94LQkLxZf9iy 2892qtpJDhTyEm9idGXX5SnkaqAsKSV5ObVA9MgcioLQejy2GEpNWqkqSHCx4anQn0sF 0iffn2SWl67e/8g7JR1G+5aexFfGxDtILBbnmRsJA6nTaOfM4dKyXr+lv9cUcAOSrYqh cwsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Kue5H9yCIZODI4T4AifLYhnGwkJ2G/QsVSJ2CiNfkS8=; b=gGLepAgJrLBoBdee7sCRoNtnLTcqoLMDZmYDwuVI2ruQYd2MNuZYkA3PD0S9plgh9K ef3bKV02hsvXV/fmtZ5exuyEfxQTikEfKSr0itKQfU0mKdzeWplGTjk/2yS1/7g1M5nU 5Utqh6vEVSJwItPMoei3HsaA1Sz6oJkkQtvl+hnlEClN7pzBwKfU6Lm6rt3FHvSnhEVi GN8m5nWmnwkh3g4fFbKIUXmhWSNs1x0FAxlud5BYEWl2v9b8pCsV//FypCgU0eWKA5Kr AQpSkuAVCqueg4gWMaq8coO4dDpFgCWODgx4qVkWQTEN5YJgWHbv+QMx5v4Yt+AVqaBU 1A+Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:18 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:18 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319849" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:17 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 07/26] x86/cet/shstk: Add Kconfig option for user-mode shadow stack Date: Mon, 19 Nov 2018 13:47:50 -0800 Message-Id: <20181119214809.6086-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introduce Kconfig option X86_INTEL_SHADOW_STACK_USER. An application has shadow stack protection when all the following are true: (1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled, (2) The running processor supports the shadow stack, (3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack. If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection. The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 25 +++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 32 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 9d734f3c8234..86fb68f496a6 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1919,6 +1919,31 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_INTEL_CET + select ARCH_HAS_SHSTK + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 88398fdf8129..0e4746814452 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -152,6 +152,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Mon Nov 19 21:47:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689529 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B224D13BB for ; Mon, 19 Nov 2018 21:54:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A23462A583 for ; Mon, 19 Nov 2018 21:54:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9592B2A6EC; Mon, 19 Nov 2018 21:54:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 272A22A583 for ; Mon, 19 Nov 2018 21:54:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD6676B1C91; Mon, 19 Nov 2018 16:54:21 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CE3426B1C93; Mon, 19 Nov 2018 16:54:21 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5E4D6B1C95; Mon, 19 Nov 2018 16:54:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 6DCBE6B1C91 for ; Mon, 19 Nov 2018 16:54:21 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id l131so21627368pga.2 for ; Mon, 19 Nov 2018 13:54:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=WFP6FoZW7tup1MGgs0fuh6+tYkyZH7kbE+iNnuvdYoU=; b=D0diIHrrl4QpYSni/H46Q3x2ZY2TCKXIp7IznC/6bNnyZvTBLq/jmMNVo1ovAMPs36 DN26X/NajYnlA9LLTplzmKoTAjzlPwk0WX0k2YRpuijity8h/ZsVtrHj0LOeuKAS83qh o1xlhiiSHlF2vH4L0VlNnn51/L8kW/cDST93XfaQJVAGBdhW6eMcxMoUmQLNm8PeCph8 kVgyQX2vuWzb5wSRaFZ0K8dCAiH/3NK5kkCmHwWz8GmD3nRCmt0FILuSLRC+npk5wiES G7QUAW5MNfDJhzKqw7hOajIPXS6MX7TJ2icwu0Wsgp1PWMWfkbOMd+zWXow+/DTJNz5b ilxg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJbDWRMsddrlW8R7n1AxfZDzJkk6d4sDouGijdqp1b0m/Oql9S3 bjcApyEALxQ9+Nuc5kHiYjKuz5GfPg1p3F4me6v0w6E7y+RBDNuqk2uQhwHSpyds+nvix88B/YL eK9ONQYu1Pvti80gffgq95y7/jsyzh+ATs4P84vCsUFsC9voWB81UCpRngD053/P4gw== X-Received: by 2002:a63:f006:: with SMTP id k6mr21776706pgh.259.1542664461102; Mon, 19 Nov 2018 13:54:21 -0800 (PST) X-Google-Smtp-Source: AJdET5fFGjqoN9K41r0e90dC23a9703Di13gOkaHwZSpcRNayPRjpUcz9PfJea8/1SUHHf4yQJLt X-Received: by 2002:a63:f006:: with SMTP id k6mr21776669pgh.259.1542664460279; Mon, 19 Nov 2018 13:54:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664460; cv=none; d=google.com; s=arc-20160816; b=WqYL05OwW4NoLDzR3jk5yk4sdvFUoWtgWk+J0SHuxB53rYch1iLFFeGC3VIfBxacZa +2R00Hjzo38R0t1XdPl8X371Wcn/RfWlARJkSQXE2yCS1RzQa1Jj2ZTP03fA/H7PDvsk EzDcZ4UIFd+xU/TphrM/77G9G8EB5Ky6p/LwWxBtp19fnSJzmAEqd7xZ3UIv4gHP4XED neqdXOaNdL7h9PH47Mgecz+ByRuiIeSnIvwWF0thKm4PayImNP2eNm1f6GdLV/wep+VM EEpl/nrpJhp3zQXOUwft16bgIzO5YX4SW03qUc23s2eWKdyZ0v7tjpk7ghxnn7qZWHJg PYaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=WFP6FoZW7tup1MGgs0fuh6+tYkyZH7kbE+iNnuvdYoU=; b=AQxQcDa+KTWLPn5Rl3fH/4UYDHLER36T2cjd/tBFmLyPqAg9xDxrWuTmX2eYkNtb9d UfeOzxSqN8wFzOcozpa3sBbjpv7MsmQx/GZeGXktBdpbhvMHvXrR1fhcR3yEaNoYeLEz SARM4oLX6B5tz/Y5WMraiITPy22lTTIMkTvXad0QSEWo0DwwT3KWofMO22S04YT4ukym 3Z2yfI91TZje5ELLZ6xVLhc+DJT+/lZl/Nv+BRTTlQL12JavtPh2SKK4xT/dwCF0Pq3P ack/UH2FSH0NZAFqIGnDctXK5390GIxDy/8QkFGoGuHn+GFpqSuQkJJEmHbnkNXb/W3Q LSlQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:20 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319853" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:18 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 08/26] mm: Introduce VM_SHSTK for shadow stack memory Date: Mon, 19 Nov 2018 13:47:51 -0800 Message-Id: <20181119214809.6086-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP VM_SHSTK indicates a shadow stack memory area. The shadow stack is implemented only for the 64-bit kernel. Signed-off-by: Yu-cheng Yu --- fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 2 files changed, 11 insertions(+) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 47c3764c469b..1ed311759dfb 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -652,6 +652,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "", #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif }; size_t i; diff --git a/include/linux/mm.h b/include/linux/mm.h index 5411de93a363..7873ac3635a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -224,11 +224,13 @@ extern unsigned int kobjsize(const void *objp); #define VM_HIGH_ARCH_BIT_2 34 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36 /* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37 /* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0) #define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1) #define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2) #define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3) #define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -266,6 +268,12 @@ extern unsigned int kobjsize(const void *objp); # define VM_MPX VM_NONE #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif + #ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE #endif From patchwork Mon Nov 19 21:47:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689531 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBFE76C5 for ; Mon, 19 Nov 2018 21:54:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD7842A583 for ; Mon, 19 Nov 2018 21:54:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B0BFE2A6EC; Mon, 19 Nov 2018 21:54:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33DA52A583 for ; Mon, 19 Nov 2018 21:54:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3961E6B1C93; Mon, 19 Nov 2018 16:54:22 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2F66B6B1C94; Mon, 19 Nov 2018 16:54:22 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19D006B1C95; Mon, 19 Nov 2018 16:54:22 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id B0E2C6B1C94 for ; Mon, 19 Nov 2018 16:54:21 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id h9so16485975pgm.1 for ; Mon, 19 Nov 2018 13:54:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=SJCNv9K8928ck0TsHD88y6FaxV4o0U2tlV6oNjQ/XIc=; b=iRK0uXHRg3tlBP4ojmNlvYCzrHKGto7wbicS25QFG8DmL6qbSEjCBEqu4BdUQ5DpOT Yo/OBY0mkXJWGkn1Q8NvuaLZPYWKvMzv2/92r8upeXSJBRbZAr+rc/BQZbb/XzbkXukL 7ODGQNbFbGUnLoHyd/6IUBhvpWZuw3EV4j1F+jNfEFaympk/18NVkUsCkpuwJm8YEeek 90eYZTsnm4lsxFBzm6I2LItjf/DIkH+Ph/9yTCAStcDuKpYn5/un8UaFLeJsoHMnin7C JqqBS5dU+cmUKwgq8FTVOr9BFQBkzbJrB4+XvPaGSdzmr8yhzKma2xZKoxAeY1bDOAPn 4Fww== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWZhJ1j9WzqJxWXNAjznacI8janMgPeLmHUh7IxWeI/AGkxNGufc WMbKeIVXVkTTpEkPNpOb/ZvXD7QxpT/iTfxG9Ukxx/FrakxAm9MdS2snAXvRDXz4rimqsJlEViB PPdxBe39m6/e8KSLUrtxEd7jFwj+nJnCAnVsXfEc59m4waHa9hJ+oOH+jef991ihEkw== X-Received: by 2002:a17:902:3064:: with SMTP id u91mr10310412plb.325.1542664461428; Mon, 19 Nov 2018 13:54:21 -0800 (PST) X-Google-Smtp-Source: AFSGD/WhGnmm3WW39vMNNXXt0Mc029Q7bla/QEtesYMmLHZ2LRg5LNKKbZlLOkjvFEdueU3l1hPJ X-Received: by 2002:a17:902:3064:: with SMTP id u91mr10310383plb.325.1542664460724; Mon, 19 Nov 2018 13:54:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664460; cv=none; d=google.com; s=arc-20160816; b=XLXd3f4nHHRmd2E7bavugXFdmC2dJctXpXiV8H0xqOdhXWJFThomqtEEXLH/cX0kdZ QzygakEMX+3fR4aXS0yLeVGhUX4StmgPs61eLUANget4YQ+IPQUl6JtPLWUjS/QzQvgP Z2ObSkMV8H2IDCuwyazEr5xh2RaB4K2YsjIrXLNQvZ5eEvIZAG8ei+WKyPzHoD8XyIUH T8xI/GohJjvTxcKLkAMFMONv/9JeKM8sg+UNAfMxS5NCYl5eJzpgiwXbwSCH69Ray1kG rtkSCZqMJbPKreNDEndUibDAhr6fr25rC7vY98WmrX+6Bcs2tkwP1YZCg81p4KQbSDhc FwMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=SJCNv9K8928ck0TsHD88y6FaxV4o0U2tlV6oNjQ/XIc=; b=Jjfpm0rFtKr8OecTHzta7eb06JgEWA/KqTLipd+TXYv7AqfL0AWcaKLvMh3jQIL6Tv Qex3A0hdbu3bYDoaVB2h9RhATTe0dw+Gldl9rrtDe0Au3oPU87zr44SwCkv5Opgl4GCg fxOLLDTCpWD6WGGkoYLmtDG96QYM6AQs7ZCPUKZRACwXOLSLdH9iqozZvHNx5fs5t2Lm +2NmqPjM+T3v8PoHpYIj9fjRgeYxOHM+jcx37YbLNBnjKuXJfeKOtim0VFkiQK+BsVPv NgEgaskOEKxsB3nuRmVec87YdGbnML0OlZq8LGvD9wbCrOa1/5SL56lC3J7kTPSL2Y+i Grfw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:20 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319857" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:19 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 09/26] mm/mmap: Prevent Shadow Stack VMA merges Date: Mon, 19 Nov 2018 13:47:52 -0800 Message-Id: <20181119214809.6086-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To prevent function call/return spills into the next shadow stack area, do not merge shadow stack areas. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 6c04292e16a7..30836512ca79 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1147,6 +1147,12 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + /* + * Do not merge shadow stack areas. + */ + if (vm_flags & VM_SHSTK) + return NULL; + if (prev) next = prev->vm_next; else From patchwork Mon Nov 19 21:47:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689533 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 580E76C5 for ; Mon, 19 Nov 2018 21:54:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 494242A583 for ; Mon, 19 Nov 2018 21:54:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C9A72A6EC; Mon, 19 Nov 2018 21:54:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 957DB2A583 for ; Mon, 19 Nov 2018 21:54:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD0B06B1C94; Mon, 19 Nov 2018 16:54:23 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D36276B1C97; Mon, 19 Nov 2018 16:54:23 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B37376B1C96; Mon, 19 Nov 2018 16:54:23 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 6ABC86B1C94 for ; Mon, 19 Nov 2018 16:54:23 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id m1-v6so24568862plb.13 for ; Mon, 19 Nov 2018 13:54:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=UaGXpYZbNKlY8aP0GwtbBAyldaUPKu9xHmVne8duuZQ=; b=oSpdOPAvNLYaQAGT7yo8EvfhUQhpK64VVAM9S5ZbvVGNCYu8emtmfreGHYf0tWAtcy XtP05RM6wUiHfmjmszN4hzt9YctpHj8kmkCo1K2TaNTl0q2/8CQNVCRDU0gonfhzWlef TxAiFfFi/0EL+HRZVsbUaZWQCcFcLuwjCxD7duuvdSmPl9REpCNLxczlk5PZFBOFZxKB E6wiyjfIJGns+HxW4LX1k+KH7A2lE33oBlaKj+o+YVmltDlwQCJm9K94FK2nAU50Uoa1 o7iHcYaIbrWoPzgL94JtK9WMcHon7mWaRPGhEFqKki+qFmTGZDqJvnxtrG4bdgQJQhaJ wYjA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJJUBsrVXPRiWo4iL63UNhYSQZuqRKJk4z9GKeqrJXsk9w66Z8y d2JYrPvirMoHg0YRr0w0dndaUJqOeflHl81IxRuDjhLI2be6cyeke1X1eXVAS6TliDLzC7smuf2 DR3rj2RlNOorSOWKBxNGBXNKA3CBd0NTO9LAbezKW9A659IIwRBnIxYak/sbVcclUNg== X-Received: by 2002:a63:2f07:: with SMTP id v7mr20400339pgv.368.1542664463077; Mon, 19 Nov 2018 13:54:23 -0800 (PST) X-Google-Smtp-Source: AJdET5dMbQlGKvy9cVAtZl10eI8iU0dWMuNuKNOVlojhSZUiPXAF/nONk82LdA0pcO/v/pbx4WlH X-Received: by 2002:a63:2f07:: with SMTP id v7mr20400292pgv.368.1542664461677; Mon, 19 Nov 2018 13:54:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664461; cv=none; d=google.com; s=arc-20160816; b=CZB/AgpGv7Vi0i5K+jbyjTus85R8NFvOg9Cu52Jj+RoPegkWNhuqTdACY/bQ+tPEs8 k2LgW6P+pKohs1zhiMn0KCrHSxuF1JVYKD1cGTAdKSV4IMQGPO016voRxi1bFPSGaeEN bRjomYVNQx4dKIC9VxQyUEw8XN6tWAXttSMgztgnVfSdOtVwcmshaCvLE/+QshCS3DJ/ /cwv3702TAWgjBKB7N7Djb5J7Sue3FkaJ/H9CAtv2duhYv92qSS09EJXMWsO0b8m4jms MKEEmPliF5MRBg/YiCA1F7fRJC/YttGJocpzhtsilpqOxSxf7sQzn0QiwX2J09n/Qbxc mPBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=UaGXpYZbNKlY8aP0GwtbBAyldaUPKu9xHmVne8duuZQ=; b=gl5PzWe+5v4Zv/EzHMIGs5Zng7BW8BlWC221b+SjmWu1M0gZ7bzKe/LTegq/waKhRm 4wBQoj7fBrPFEXHKFTapaitT3dwN1gIrU4aJJMzQIpmmqgAppUdeMhnscTkyDaMNgInD xM8ZSoq5HfB7Se3Ck4rdjibRc5WKJEMlJHHGMOI88PXcPcgCH0DctTh38mMpIto59zXP LPH9C1AHpJgZuMuhDFE5JJrDfZbeQ8zcdTd2ySK2bhFJhxRT3ezczIBtmKXnMgExZpSc UGtjc+PPH5NT0B874r9r9JVG45qpB9vkSnFMC0yHKiAz1RPUYteRMi81c48GiBEeVgB5 m6jQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:21 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319861" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:20 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 10/26] x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW Date: Mon, 19 Nov 2018 13:47:53 -0800 Message-Id: <20181119214809.6086-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Before introducing _PAGE_DIRTY_SW for non-hardware, memory management purposes in the next patch, rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 40616e805292..be71584bbd9a 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -315,7 +315,7 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { - return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) @@ -389,7 +389,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd) static inline pmd_t pmd_mkdirty(pmd_t pmd) { - return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pmd_t pmd_mkdevmap(pmd_t pmd) @@ -443,7 +443,7 @@ static inline pud_t pud_wrprotect(pud_t pud) static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); + return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 106b7d0e2dae..7b6a7f088f53 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -15,7 +15,7 @@ #define _PAGE_BIT_PWT 3 /* page write through */ #define _PAGE_BIT_PCD 4 /* page cache disabled */ #define _PAGE_BIT_ACCESSED 5 /* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6 /* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6 /* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7 /* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7 /* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8 /* Global TLB entry PPro+ */ @@ -45,7 +45,7 @@ #define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT) #define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD) #define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW) #define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE) #define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL) #define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1) @@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3) #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED) #else #define _PAGE_KNL_ERRATUM_MASK 0 #endif @@ -112,9 +112,9 @@ #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) #define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * Set of bits not changed in pte_modify. The pte's @@ -123,7 +123,7 @@ * pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) @@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED) #define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL) #define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX) #define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW) @@ -187,7 +188,7 @@ enum page_cache_mode { #define _PAGE_ENC (_AT(pteval_t, sme_me_mask)) #define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC) #define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER) #define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC) diff --git a/arch/x86/kernel/relocate_kernel_64.S b/arch/x86/kernel/relocate_kernel_64.S index 11eda21eb697..e7665a4767b3 100644 --- a/arch/x86/kernel/relocate_kernel_64.S +++ b/arch/x86/kernel/relocate_kernel_64.S @@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 4555077d69ce..7bbb8b26e901 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -5858,7 +5858,7 @@ static int init_rmode_identity_map(struct kvm *kvm) /* Set up identity-mapping pagetable for EPT in real mode */ for (i = 0; i < PT32_ENT_PER_PAGE; i++) { tmp = (i << 22) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r = kvm_write_guest_page(kvm, identity_map_pfn, &tmp, i * sizeof(tmp), sizeof(tmp)); if (r < 0) From patchwork Mon Nov 19 21:47:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689537 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CE2C13BB for ; Mon, 19 Nov 2018 21:54:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E62C2A589 for ; Mon, 19 Nov 2018 21:54:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 41FEA2A583; Mon, 19 Nov 2018 21:54:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B03C2A583 for ; Mon, 19 Nov 2018 21:54:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4851E6B1C97; Mon, 19 Nov 2018 16:54:25 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3BAEF6B1C98; Mon, 19 Nov 2018 16:54:25 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23A046B1C99; Mon, 19 Nov 2018 16:54:25 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id B818F6B1C97 for ; Mon, 19 Nov 2018 16:54:24 -0500 (EST) Received: by mail-pf1-f199.google.com with SMTP id 68so21654829pfr.6 for ; Mon, 19 Nov 2018 13:54:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GAB0KUWD6DoK6NycwZ9SSWimx0j7ZnfH89zzkr4gf44=; b=i7O7TktgjGgVRbgh3HsUtM4QD3ZZIs4bt6i3gqFGR9hEyAhu5bM1hxGB2rDu1/f9kU Okfp6tFxEEWuIYqBf7YDAwjsvrZkW3xO65g77GSSPVsEbccbKSxD+opYfU7sVLkgeFIG yRs8W0sv1C1RYxfvzkAZw/NHSeGb7fW8qpRjZKRfLIaPAjZEPc+phf1puTyP21WGXsue aijE2daHNAe87W91kIvXKNREWzmT7GlO/RKtldwCRao3t4I534CorIKmsOFz7IaHRooP Q7pptSlAKxx6/hQZecjCf508mXG9kZMny6yS8b3OpgD9bCwHTNZ97QONrH4FBGbnd0J4 zbeA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJfsR4FKVLxj17sB2hqP/eOT1HXwcO761Dajo4Dbqag2LNNHDBx pu63y7D42erm4ZigzMC0BHH+8Y7tlc9HADLhmTnEP9aUb+YlQDT1CnFB/+osqgU08NrZjmlwI7c TezYbHUH4bxyklGgvLpzMO4Kpo0WzApSups6DV57Pr8Y/6LOJb1rxGPYoldmQmETvxQ== X-Received: by 2002:a63:f0c:: with SMTP id e12mr21412139pgl.274.1542664464390; Mon, 19 Nov 2018 13:54:24 -0800 (PST) X-Google-Smtp-Source: AJdET5clUcZxNe/V6N6h9AEP9WojAtwq6EZR/s2rqG8+f8rXqQEI1ntUgcYUCRByft3+v8crwcoP X-Received: by 2002:a63:f0c:: with SMTP id e12mr21412068pgl.274.1542664462665; Mon, 19 Nov 2018 13:54:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664462; cv=none; d=google.com; s=arc-20160816; b=mRhlEyHjxpkmojtpHS3Zdis5toBCWoBKJkuYU3agtTeTaxjjxcXSNFYpFm7T8nodVW QS1mssuKPvYHvPRkp/UMZnqYS3gktTjpWGqRW1QCBezBl5yHJQluXwkkeXU4g41awRAP m/cHEpgijxlDce0O5QO+VQHGY99WJQulrmF8CcyxeBeB+3+rwLw9p1FhcSf+9G2uYysu 39FnJRO1f1Cdotmksu+2p46oXdKH2Bb9lXUCi0G6+jEZWpC72lUqA9WhstP+aD1QOYB4 AgvDB7TKTjjntI2jRXjyfqHIF3fy5eTXN3j7cI4D2w2N8A03gZGpUozBR1d5J4Hy7BCN czRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=GAB0KUWD6DoK6NycwZ9SSWimx0j7ZnfH89zzkr4gf44=; b=i0t1SN1YDdxXlOzM7X+dtsqw/xVQ7ElLQ1XBHuugICc710oRoSaSYMLgs7f5wktfyt Iutj8lC4WxdDy5RFelBH/G9qEUyLw6Cx7n914LC19HtoPCgq+6F6gS4PDQ3ffNjTOFfg IzNcBBjMK9xVBuxGyVWbxjB3yiJL+HPsIKWd3k/mkFFcL/XZIh/DB5OZ1r2T2hKqygkI zLtkqNz7ujDsmqolT83tlrjExbq3czL3gigtMwP9BEs79APLfowz995TzKbzaDrP9aKz Wa/sMsR6xwbd6XhAU7+G7/12GOGdS/kW1s0pAcw2UNXXTr/DfhKSxQPlQRF4saEBdt2p JpiQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:22 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:22 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319866" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:21 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 11/26] x86/mm: Introduce _PAGE_DIRTY_SW Date: Mon, 19 Nov 2018 13:47:54 -0800 Message-Id: <20181119214809.6086-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A RO and dirty PTE exists in the following cases: (a) A page is modified and then shared with a fork()'ed child; (b) A R/O page that has been COW'ed; (c) A SHSTK page. The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings: Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW) Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs. When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 21 ++++- 2 files changed, 128 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index be71584bbd9a..db4b9d22d2f7 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -118,9 +118,9 @@ extern pmdval_t early_pmd_flags; * The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty(pte_t pte) +static inline bool pte_dirty(pte_t pte) { - return pte_flags(pte) & _PAGE_DIRTY; + return pte_flags(pte) & _PAGE_DIRTY_BITS; } @@ -142,9 +142,9 @@ static inline int pte_young(pte_t pte) return pte_flags(pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty(pmd_t pmd) +static inline bool pmd_dirty(pmd_t pmd) { - return pmd_flags(pmd) & _PAGE_DIRTY; + return pmd_flags(pmd) & _PAGE_DIRTY_BITS; } static inline int pmd_young(pmd_t pmd) @@ -152,9 +152,9 @@ static inline int pmd_young(pmd_t pmd) return pmd_flags(pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty(pud_t pud) +static inline bool pud_dirty(pud_t pud) { - return pud_flags(pud) & _PAGE_DIRTY; + return pud_flags(pud) & _PAGE_DIRTY_BITS; } static inline int pud_young(pud_t pud) @@ -293,9 +293,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) return native_make_pte(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + if (pte_flags(pte) & from) + pte = pte_set_flags(pte_clear_flags(pte, from), to); + return pte; +} +#else +static inline pte_t pte_move_flags(pte_t pte, pteval_t from, pteval_t to) +{ + return pte; +} +#endif + static inline pte_t pte_mkclean(pte_t pte) { - return pte_clear_flags(pte, _PAGE_DIRTY); + return pte_clear_flags(pte, _PAGE_DIRTY_BITS); } static inline pte_t pte_mkold(pte_t pte) @@ -305,6 +319,7 @@ static inline pte_t pte_mkold(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pte_clear_flags(pte, _PAGE_RW); } @@ -315,9 +330,24 @@ static inline pte_t pte_mkexec(pte_t pte) static inline pte_t pte_mkdirty(pte_t pte) { + pteval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + pte_write(pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk(pte_t pte) +{ + pte = pte_clear_flags(pte, _PAGE_DIRTY_SW); return pte_set_flags(pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw(pte_t pte) +{ + return pte_flags(pte) & _PAGE_DIRTY_HW; +} +#endif + static inline pte_t pte_mkyoung(pte_t pte) { return pte_set_flags(pte, _PAGE_ACCESSED); @@ -325,6 +355,7 @@ static inline pte_t pte_mkyoung(pte_t pte) static inline pte_t pte_mkwrite(pte_t pte) { + pte = pte_move_flags(pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pte_set_flags(pte, _PAGE_RW); } @@ -372,6 +403,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) return native_make_pmd(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + if (pmd_flags(pmd) & from) + pmd = pmd_set_flags(pmd_clear_flags(pmd, from), to); + return pmd; +} +#else +static inline pmd_t pmd_move_flags(pmd_t pmd, pmdval_t from, pmdval_t to) +{ + return pmd; +} +#endif + static inline pmd_t pmd_mkold(pmd_t pmd) { return pmd_clear_flags(pmd, _PAGE_ACCESSED); @@ -379,19 +424,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd) static inline pmd_t pmd_mkclean(pmd_t pmd) { - return pmd_clear_flags(pmd, _PAGE_DIRTY); + return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS); } static inline pmd_t pmd_wrprotect(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pmd_clear_flags(pmd, _PAGE_RW); } static inline pmd_t pmd_mkdirty(pmd_t pmd) { + pmdval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pmd_flags(pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk(pmd_t pmd) +{ + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY_SW); return pmd_set_flags(pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_DIRTY_HW; +} +#endif + static inline pmd_t pmd_mkdevmap(pmd_t pmd) { return pmd_set_flags(pmd, _PAGE_DEVMAP); @@ -409,6 +471,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd) static inline pmd_t pmd_mkwrite(pmd_t pmd) { + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pmd_set_flags(pmd, _PAGE_RW); } @@ -426,6 +489,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) return native_make_pud(v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + if (pud_flags(pud) & from) + pud = pud_set_flags(pud_clear_flags(pud, from), to); + return pud; +} +#else +static inline pud_t pud_move_flags(pud_t pud, pudval_t from, pudval_t to) +{ + return pud; +} +#endif + static inline pud_t pud_mkold(pud_t pud) { return pud_clear_flags(pud, _PAGE_ACCESSED); @@ -433,17 +510,22 @@ static inline pud_t pud_mkold(pud_t pud) static inline pud_t pud_mkclean(pud_t pud) { - return pud_clear_flags(pud, _PAGE_DIRTY); + return pud_clear_flags(pud, _PAGE_DIRTY_BITS); } static inline pud_t pud_wrprotect(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); return pud_clear_flags(pud, _PAGE_RW); } static inline pud_t pud_mkdirty(pud_t pud) { - return pud_set_flags(pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); + pudval_t dirty = (!IS_ENABLED(CONFIG_X86_INTEL_SHADOW_STACK_USER) || + (pud_flags(pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + + return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY); } static inline pud_t pud_mkdevmap(pud_t pud) @@ -463,6 +545,7 @@ static inline pud_t pud_mkyoung(pud_t pud) static inline pud_t pud_mkwrite(pud_t pud) { + pud = pud_move_flags(pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW); return pud_set_flags(pud, _PAGE_RW); } @@ -594,19 +677,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK; val = flip_protnone_guard(oldval, val, PTE_PFN_MASK); + if ((pte_write(pte) && !(pgprot_val(newprot) & _PAGE_RW))) + return pte_move_flags(__pte(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); return __pte(val); } -static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) -{ - pmdval_t val = pmd_val(pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; - val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); - return __pmd(val); -} - /* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modify static inline pgprot_t pgprot_modify(pgprot_t oldprot, pgprot_t newprot) @@ -1158,6 +1234,19 @@ static inline int pmd_write(pmd_t pmd) return pmd_flags(pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) +{ + pmdval_t val = pmd_val(pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK); + if ((pmd_write(pmd) && !(pgprot_val(newprot) & _PAGE_RW))) + return pmd_move_flags(__pmd(val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); + return __pmd(val); +} + #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h index 7b6a7f088f53..6a2b42490fb2 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -23,6 +23,7 @@ #define _PAGE_BIT_SOFTW2 10 /* " */ #define _PAGE_BIT_SOFTW3 11 /* " */ #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57 /* available for programmer */ #define _PAGE_BIT_SOFTW4 58 /* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ @@ -34,6 +35,7 @@ #define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1 #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5 /* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ @@ -109,6 +111,21 @@ #define _PAGE_DEVMAP (_AT(pteval_t, 0)) #endif +/* + * _PAGE_DIRTY_HW: set by the processor when a page is written. + * _PAGE_DIRTY_SW: a spare bit tracking a written, but now R/O page. + * [R/W + _PAGE_DIRTY_HW] <-> [R/O + _PAGE_DIRTY_SW]. + * _PAGE_SOFT_DIRTY: a spare bit used to track written pages since a time point + * set by the system admin; see Documentation/admin-guide/mm/soft-dirty.rst. + */ +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) + #define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE) #define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ @@ -122,9 +139,9 @@ * instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY | _PAGE_DEVMAP) #define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE) /* From patchwork Mon Nov 19 21:47:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689535 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 585166C5 for ; Mon, 19 Nov 2018 21:54:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A4162A583 for ; Mon, 19 Nov 2018 21:54:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E61D2A6EC; Mon, 19 Nov 2018 21:54:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA7E42A583 for ; Mon, 19 Nov 2018 21:54:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E98CF6B1C96; Mon, 19 Nov 2018 16:54:24 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF48E6B1C98; Mon, 19 Nov 2018 16:54:24 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6CE86B1C99; Mon, 19 Nov 2018 16:54:24 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 84E876B1C96 for ; Mon, 19 Nov 2018 16:54:24 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id v79so4405701pfd.20 for ; Mon, 19 Nov 2018 13:54:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=d/wTFUPAs+KlOihPh9SXXZ7HI7k2d1EFWCjRpMRif20=; b=QfTlf48FJW00K6ftWLUMDvBcHayDU6dP7hsG5u8FuqtFvXbsVJczqqoOlDal1yktVV 78//Ya3YzYz0jSQSU9by7VZ1z/a+KXXGlOKdaTsJeG9LvnBWryjf6LWOeuCRb9F6IKFa J25nCIz78uXfz4z4xdRR99u2l8YGD5bfS9Uga528IS1iRbbhxXivglLYdh7QVCA6eeLn Cv0aV026PaAwtZ1uf0vDtsf3kp8BoeExYxqGohCMgub4N4jezRTgGJ6bmsAj5FJwI4Up nVSb4ujLDCPCIMZO73RS8kL4mCKfrxj92bPYo4cxrKE7T5Va4ZoQ26T+q7hRpzL7zj2J DknQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJI/0dQet26a6XeNAhKULnKLdovf38YvJ8A2OTIMZ2LJuTgnsGx 7UdNxx6rcQL/SnqhN1yrG4lcq9dnodhbJEq9eeaFdtv9huOnFEvL2+6BcsTIpPDpMpMS7/GZkJl K2vSomWzgQbmyYGki5dbZEvFuM1nc3MnglQJVRX4NPJmivw4NLZ6USYpg+/aVqF3aRw== X-Received: by 2002:a17:902:3e3:: with SMTP id d90-v6mr24008958pld.118.1542664464224; Mon, 19 Nov 2018 13:54:24 -0800 (PST) X-Google-Smtp-Source: AJdET5fzfLwndU5U6d47kjCimP7EPR5nT6wgluuVAWPfOfcGZK+1tm1htbpa9qwCbi16PaAd5VHr X-Received: by 2002:a17:902:3e3:: with SMTP id d90-v6mr24008914pld.118.1542664463449; Mon, 19 Nov 2018 13:54:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664463; cv=none; d=google.com; s=arc-20160816; b=G/FT9usYgHU+Mhqul8hr5p6JCrc0h1NuHmJUdNwis9VbfW0yjN/Gsx0k+oKZ720nbS OZhIoI5rG9UCB32nxWho1s17z5b+D1MvYXhsx8VLmP6RHkxpdFROm2XKmykBDXb7QHeT HHU8IT60AUfSvf7k+54msZtzXWyEFAU+78jrdPx+sIGMRSVLI1jwUvrY8y43eFabLZ2b IQ2MUPa52qSbclvCL43SFSvUs4CP64Hjn6zdh/9a1/brhAEh1TfXtuqgetMgmY3l3q0h vCuHN0FUnvPKnUafV71Y+2J2iCl/5GM2FNo7ozzr2V1G9zsAomeQC+8lQKeHeFicdrum 3zSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=d/wTFUPAs+KlOihPh9SXXZ7HI7k2d1EFWCjRpMRif20=; b=Iu8ZE/fz7UcS+bAgClBMP7ScZXh8/CjiXs7lVzEyueySsoNjFBYcB4n+dq2GKB6uTq 7UNJD9C2kOj9CMrjk115pqs28YABdvaGAFp54tGZ324PUbAOHklwff1ktDZVFV0BenxP 4mh+H+svRzpIOo1zT3qc9iSgNNaDcYP569iUhBnNJNmW0hzKqhIrXeH5Cu6ZVFnH05yD g5+szLZhvLOX6b3clWKGgNP/eakn2k55+skJhKHkyxpsu3USrC+ToDrI5/5BREhCHA4n BnawK+5YSH2+fI70e693VZkY0TA8O2nFiem8W2oRiuC65EnKAEHqGB5KSZrefIeFKnmp cq4g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:23 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:23 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319872" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:22 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 12/26] drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS Date: Mon, 19 Nov 2018 13:47:55 -0800 Message-Id: <20181119214809.6086-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Update _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry(). In order to support Control-flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW. Signed-off-by: Yu-cheng Yu --- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 58e166effa45..6e94e27d810e 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -1174,7 +1174,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, } /* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse(se); ops->clear_ips(se); From patchwork Mon Nov 19 21:47:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689539 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AF1C13BB for ; Mon, 19 Nov 2018 21:54:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D2372A6EE for ; Mon, 19 Nov 2018 21:54:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3F06A2A6F7; Mon, 19 Nov 2018 21:54:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A17202A589 for ; Mon, 19 Nov 2018 21:54:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 260746B1C98; Mon, 19 Nov 2018 16:54:26 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1EA8A6B1C99; Mon, 19 Nov 2018 16:54:26 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E69986B1C9A; Mon, 19 Nov 2018 16:54:25 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 9E4F86B1C98 for ; Mon, 19 Nov 2018 16:54:25 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id 68so21654855pfr.6 for ; Mon, 19 Nov 2018 13:54:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=4lNjvzsO3J8a39KYcX74z0FBTxk0UzpiYaAAYtfEg/c=; b=AgaWm7fWvUhKkKxseycwFW3mnZ4X4Z2qr0oW5tBD5vltwQEq6wt9D9Ec4cwcWLagBy /AKLc+4CzTfkQtluJ5XdSMadFYyaEFB4hd4EqZ+eyYVy1SF1/RTYJjZd9/T2RsFL5M2G DMIP5tiu+15as4aVb/XsW3O4Z9YpO3g1jmca6+DDFeedmyk3DPmCXVkUpAJmb0/JZAlF m+b9AsTv5zncElgFuxNo9XevldCZu+RjPLxl1e8zBzoapt6bxHhR0OktIrkWEyHdINcF Ra8nWmOyt+WgZxj0cSqmFHezlVMNtF+Qc29QthBohDjchXz/VuIw5HAzocafsIKrm+Re k4uQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJ202EM7LAvHXyCKLiiORqqyJ8JkHVdHz1mQf96N6LVUe41vEYc V5HfiOutRveEqWsuT8lQekB1h7E7vUm6PZJruR5jlfXD8PFmO3Kv767ScBq2t163eDwW+r0Z1Zn yAagW4ft2WsFyyvl6+P+qtC01MiZPC1VgGYU97ytvlSI9XQBH6xEWUlB09ORJBfPVLA== X-Received: by 2002:a63:a51b:: with SMTP id n27mr22090114pgf.17.1542664465309; Mon, 19 Nov 2018 13:54:25 -0800 (PST) X-Google-Smtp-Source: AJdET5fwNeYgbcZgnuxcS5IRHljXF7rgpy8GHhDFv6lspszt21MMy8VDXGxvY1H0cLMZ6iOqUfj9 X-Received: by 2002:a63:a51b:: with SMTP id n27mr22090081pgf.17.1542664464449; Mon, 19 Nov 2018 13:54:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664464; cv=none; d=google.com; s=arc-20160816; b=dSYi70WVmuVzutuOFAvx4MohFToJbEGhmBe+N/JAfSKYXRMdZGOQUHBlVRmyKHx2zE zzeOJLQ9no4k4nNkMltwwcMBsyfKUXgd5d4QWomFX2w8qtIz67fmLW0OgkHh6dozFoUh rA3fr6xzMuwDDB9c+xVSbViyY477UGLCad2iS4F5SC+TjjKvVWTJIe4f9wThlaFqQjm0 UJ+OOVcMadobm2j6kLmOWYEiQ48ZQKnQjRtVI0VzruDLdQagw1QUObolCPGZ/5z+XZCO XIpjo9IfZh7Z9oRjG/iquzmn2MwL4Lw293XoLuAepKeqpm2gw48BlzHJsF1anmDLlZva Flaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=4lNjvzsO3J8a39KYcX74z0FBTxk0UzpiYaAAYtfEg/c=; b=KMN7E1Q6Y6qFwoGHeMXNjuW1CDZ8ZXmfftwTGzstGxjZAgJbyxVrjteBMJd70h0RIg /dt7p0m9K5Kdqyo7TKuiURdNfVKi3VQhurJbOdOgg3129PqjyhjlLQVIu/DBVG7YWI0G p2VD1Q5Q826x5yplgWHuezE7Ni8yIs0lmBSt6t/RqQSclwwZJymjEll6z8wR/QyiB3MB smVJteklG8pgzu52QBxCxhiYLCQuFFDMAzGfDl+fyeckFX3Z/G30nijU6yQoFZUt9k9H qt1Af5SceZeBTWNLcb1avlRTRrSteLArrImRJIHk3K1ZC9xP4BZO6yFFLOilIjsUes6g V2Dg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:24 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319875" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:23 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 13/26] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW Date: Mon, 19 Nov 2018 13:47:56 -0800 Message-Id: <20181119214809.6086-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When Shadow Stack is enabled, the [R/O + PAGE_DIRTY_HW] setting is reserved only for the Shadow Stack. Non-Shadow Stack R/O PTEs use [R/O + PAGE_DIRTY_SW]. When a PTE goes from [R/W + PAGE_DIRTY_HW] to [R/O + PAGE_DIRTY_SW], it could become a transient Shadow Stack PTE in two cases. The first case is that some processors can start a write but end up seeing a read-only PTE by the time they get to the Dirty bit, creating a transient Shadow Stack PTE. However, this will not occur on processors supporting Shadow Stack therefore we don't need a TLB flush here. The second case is that when the software, without atomic, tests & replaces PAGE_DIRTY_HW with PAGE_DIRTY_SW, a transient Shadow Stack PTE can exist. This is prevented with cmpxchg. Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/pgtable.h | 58 ++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index db4b9d22d2f7..cf0c50ef53d8 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1202,7 +1202,36 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pte_t new_pte, pte = READ_ONCE(*ptep); + + /* + * Some processors can start a write, but end up + * seeing a read-only PTE by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PTE which looks like a Shadow Stack PTE. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PTE and flush the + * TLB would be needed. + * + * When changing a writable PTE to read-only and + * if the PTE has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PTE is + * not a valid Shadow Stack PTE. + */ + do { + new_pte = pte_wrprotect(pte); + new_pte.pte |= (new_pte.pte & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pte.pte &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(ptep, &pte, new_pte)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +#endif } #define flush_tlb_fix_spurious_fault(vma, address) do { } while (0) @@ -1265,7 +1294,36 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + pmd_t new_pmd, pmd = READ_ONCE(*pmdp); + + /* + * Some processors can start a write, but end up + * seeing a read-only PMD by the time they get + * to the Dirty bit. In this case, they will + * set the Dirty bit, leaving a read-only, Dirty + * PMD which looks like a Shadow Stack PMD. + * + * However, this behavior has been improved and + * will not occur on processors supporting + * Shadow Stacks. Without this guarantee, a + * transition to a non-present PMD and flush the + * TLB would be needed. + * + * When changing a writable PMD to read-only and + * if the PMD has _PAGE_DIRTY_HW set, we move + * that bit to _PAGE_DIRTY_SW so that the PMD is + * not a valid Shadow Stack PMD. + */ + do { + new_pmd = pmd_wrprotect(pmd); + new_pmd.pmd |= (new_pmd.pmd & _PAGE_DIRTY_HW) >> + _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW; + new_pmd.pmd &= ~_PAGE_DIRTY_HW; + } while (!try_cmpxchg(pmdp, &pmd, new_pmd)); +#else clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp); +#endif } #define pud_write pud_write From patchwork Mon Nov 19 21:47:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689541 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7A236C5 for ; Mon, 19 Nov 2018 21:54:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA3302A583 for ; Mon, 19 Nov 2018 21:54:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D5AC2A6F3; Mon, 19 Nov 2018 21:54:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7AE3E2A589 for ; Mon, 19 Nov 2018 21:54:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EDDC6B1C99; Mon, 19 Nov 2018 16:54:27 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2F8F66B1C9A; Mon, 19 Nov 2018 16:54:27 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19C726B1C9B; Mon, 19 Nov 2018 16:54:27 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id AF2246B1C99 for ; Mon, 19 Nov 2018 16:54:26 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id t22so641579plo.10 for ; Mon, 19 Nov 2018 13:54:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ihDwmQNOBYtcVZgOMT8iFE8SC6HjwQGnvLkO9zWURcg=; b=tCrh4bj7v1bc+ieTT570NK2sSrJRLLlA8yRVYOg7WTE5DqhvPGGQS7o5BwsQX2uRhj KGhYGXEvkr6mfldjDVn8iQtn9xRVMZ9ew5Lqqk3vwv9vLb29BAOMhhe5nZ8P8k6uWg5Y 18+bCYFHcyjL1U5z/GBek/Czj2CK1ZJfg670Sy3a6cLAs35HEly74HPqr1KPPoucmGT1 dTUoJB8myseF5lxKdyU/9nQ3D9nq4incd9VMRV1Pw3YZH6qJTdPSB4sGaiS3N5MdQZBs 5xomhy0Mi1pPOWwsFSL0QQz3glFZctfam99ykweGMgqJGLWvgOE9mhUB+gUIxc8qWA56 5ukA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWaaa5Ac3fnFKIdJanGp55h5fzfv5G/MKhODTZZQtsEO8j9euDhK eF0fWy2osKgdK+FoK8shQ/ZxQS6n4kmPnexsT1gzO+YQG0Y6UNW8+XVadxguBw61jr3Kjd/V85s 67QFpt2nQSsQ79ZtCCKapsXNxMNze1FYy217T7QQCHvjuJB4hAsXmG5rUP7/DjsjYtw== X-Received: by 2002:a17:902:4222:: with SMTP id g31mr4437263pld.240.1542664466399; Mon, 19 Nov 2018 13:54:26 -0800 (PST) X-Google-Smtp-Source: AFSGD/UGgePqsShNe95/8M7TloWXiSkgxqG/4m8esNf/5iKUQYFl0oVtS5HBuC+LE3n5Q4bCJdGn X-Received: by 2002:a17:902:4222:: with SMTP id g31mr4437226pld.240.1542664465302; Mon, 19 Nov 2018 13:54:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664465; cv=none; d=google.com; s=arc-20160816; b=dYd/IDhzh9pmi8Rj0UsBF7C7qRQsOm/GTx3ROU/VXe2P3rF/XB7XBECccyM0AEAbJr 8QKctCV/dCPnmeV2FFO52BTnGmv5SrPut6Q/fsodXpYoIm1YyBYSU1gWry+6DkvbwM+y gGDqsohslaXJ2OwRM427NObUlObgob9+ybDvoI93gBo50V3yKQW/ILv0uWSDHMzRsDAd +RWJfFcfnZDJPcA64c71UESUGL6pBvFVbfvun43VXusw53buqCdsGTUqAfVuxf4szZdd ZnxqxyiFkQ+hqxtNFCH9YHkQj7rceeugxF5OXllZ9UYepjVvIpv5faKOsbfM3gqDS7bk JfLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ihDwmQNOBYtcVZgOMT8iFE8SC6HjwQGnvLkO9zWURcg=; b=ma5ufZ+rfo1oMbO5jp/R7xXmTO/9ZdSXWdqIW37ohQY1VTyt+7SJi+SV465Je4h3to OQ3Tpad7mq2MEl4AcMcwH+e86fRe/Yh59smosA1ptLyqwrtq94jV787XpdV5NKfJaxTh trnCI7s6mOSkeRcmpRcKRYgIAMlbN3oL9wmvgL9vUABBeXjf3117f30RH4OMevlbmKH/ Cwjx+LmbhQV+YummLnb1s+bOLLClU2bmcKDFiL9iQhB3C3amcGU/WBjEulHpGQ6tTg/C /ANh3rNfxdJBC2kiIfC3J92bA4gR0EOT+1yA20ThaAQdZyrAhpDn1CMuPrIc0nvbnpPY qpKg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:25 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319879" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:23 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 14/26] x86/mm: Shadow stack page fault error checking Date: Mon, 19 Nov 2018 13:47:57 -0800 Message-Id: <20181119214809.6086-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set. In access_error(), verify a shadow stack page fault is within a shadow stack memory area. It is always an error otherwise. For a valid shadow stack access, set FAULT_FLAG_WRITE to effect copy-on-write. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h index 20841d301e93..50e8a670624a 100644 --- a/arch/x86/include/asm/traps.h +++ b/arch/x86/include/asm/traps.h @@ -157,6 +157,7 @@ enum { * bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access + * bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code { X86_PF_PROT = 1 << 0, @@ -165,5 +166,6 @@ enum x86_pf_error_code { X86_PF_RSVD = 1 << 3, X86_PF_INSTR = 1 << 4, X86_PF_PK = 1 << 5, + X86_PF_SHSTK = 1 << 6, }; #endif /* _ASM_X86_TRAPS_H */ diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 71d4b9d4d43f..c3368fed706c 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1117,6 +1117,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign)) return 1; + /* + * Verify X86_PF_SHSTK is within a shadow stack VMA. + * It is always an error if there is a shadow stack + * fault outside a shadow stack VMA. + */ + if (error_code & X86_PF_SHSTK) { + if (!(vma->vm_flags & VM_SHSTK)) + return 1; + return 0; + } + if (error_code & X86_PF_WRITE) { /* write, present and write, not present: */ if (unlikely(!(vma->vm_flags & VM_WRITE))) @@ -1313,6 +1324,13 @@ void do_user_addr_fault(struct pt_regs *regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); + /* + * If the fault is caused by a shadow stack access, + * i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set + * FAULT_FLAG_WRITE to effect copy-on-write. + */ + if (sw_error_code & X86_PF_SHSTK) + flags |= FAULT_FLAG_WRITE; if (sw_error_code & X86_PF_WRITE) flags |= FAULT_FLAG_WRITE; if (sw_error_code & X86_PF_INSTR) From patchwork Mon Nov 19 21:47:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC74813BB for ; Mon, 19 Nov 2018 21:54:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEAE32A583 for ; Mon, 19 Nov 2018 21:54:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A29572A6F3; Mon, 19 Nov 2018 21:54:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 135442A583 for ; Mon, 19 Nov 2018 21:54:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0B466B1C9B; Mon, 19 Nov 2018 16:54:27 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E6A586B1C9C; Mon, 19 Nov 2018 16:54:27 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBB5C6B1C9D; Mon, 19 Nov 2018 16:54:27 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 836336B1C9B for ; Mon, 19 Nov 2018 16:54:27 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id k14-v6so24583098pls.21 for ; Mon, 19 Nov 2018 13:54:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=O92AXCbrqBSxJV0ao/5hXiZk219rpE4NX9OaI3z1mq4=; b=cpMSWs8UCW0t3yg/Zrp0HGGbdCiDP6lrhTFxsC9bzxh2HRyEGASTVis7S/MTpDJmfq NWr32lilqto4teLycoFugocSMHvIlNV0pEDODNEdQZA/IfiuVrPXfAP/2RhzhE2APmjS +7jXT+O70KAOu14N0f/T1vb2e/Z2JYZ/ZgOY52+YKtnS8I+to//XLgBsKgY2akbd7lTH WHbgr/aFdXl+daweXI5jkbTzmKdfPy5VvkXaGKJMi5Otn1o0LayXKdRuWWBteUhzYjjF shxXIOFCyUQDRqjDa8+VTfn2xk3LNCsJs2u3tTAI2ECDkxxTemiCHTeJy3XTBXTbx2I2 J0EA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gKkENt4fazyIPOJdJ9Y1whqe/dMhaUIV73gKQlBNTancXOp5iLB CUZtkHV5uWTENkjNjuqMWrOw8jAMZP7kmEH22ILPL/ItdzamWIsfPYJpIGhm+VQc4HHL5evL1Cn U+4Sp9omUl/2tPzM/zxqll5xn3fcCKXNrhuyM2zt2ZE+Zt6EbIJVKJ5YPlGoOj74MvA== X-Received: by 2002:a63:9712:: with SMTP id n18mr21317081pge.295.1542664467185; Mon, 19 Nov 2018 13:54:27 -0800 (PST) X-Google-Smtp-Source: AJdET5fAcfVXD2povKP3D0a0bFJgec/+9Spgrs23DDG4okP77cRnGOSXeNyQFpPodYQil5Dpxxdn X-Received: by 2002:a63:9712:: with SMTP id n18mr21317050pge.295.1542664466439; Mon, 19 Nov 2018 13:54:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664466; cv=none; d=google.com; s=arc-20160816; b=x2n0QfLEZRxdy1gVZMLZwCA1s/WEtgRMIW20q9QY0X58HbCWhjiYpwJqheo+XfF26W jPNFujeBOcIPAfWZU/5JS6AFemSBdb2A1tHQw52fsQH81Nazp/RmC6BPXsvfsvs3D16S HSSaKPGXTAc9AE2jCqdC52YIqZHInX8/Sa645pRiljSrPeWxWD7PNB1O5Ab2wdnGZ6Y9 9RxUzWIqWDJ9om9f9wye43QUdCVDiYUNnWTDMVe4EDttdfLl3QkP3cOMjIT2cNwiPq7N 1/vZb65dqFL+wXwf7pKt3rN0Di7yyk1EQWUztIH6WP1crqi8+D75tchkmUIQ80TrCioX k/cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=O92AXCbrqBSxJV0ao/5hXiZk219rpE4NX9OaI3z1mq4=; b=gdn0Ny93tSNFqf8Qar88tLsliytaDcK5ptg3UMI2kLSryAnXYdsmibF+akv1fdxgVG lXAQq4T5yxQJ2xt/0kghaaDBmgKs864HW+0J+vVJYI5oZ+OqxQrYvydnSiUI2hjorKRR 9OpHOnEJXfvFamvKyuKJAboIBqVH1v9wobi7YXQ71/ryZSsTcyzbTbNBkbs6nfmA8xbQ YkeiS93tvOk/QgLWyw3bhYYErqRMK1SqYPeAPYst/eLPQrzFOe70FdIg8fUpI+Qym08C Ktzqi5S5HWZn0b4HguBIYcXSmj5P81d4ZDVdCT3h7FarsMq1poCSfMkk2Q0mnAbvJIms vt7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o12-v6si32492543plg.114.2018.11.19.13.54.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:26 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319883" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:24 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 15/26] mm: Handle shadow stack page fault Date: Mon, 19 Nov 2018 13:47:58 -0800 Message-Id: <20181119214809.6086-16-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When a task does fork(), its shadow stack (SHSTK) must be duplicated for the child. This patch implements a flow similar to copy-on-write of an anonymous page, but for SHSTK. A SHSTK PTE must be RO and dirty. This dirty bit requirement is used to effect the copying. In copy_one_pte(), clear the dirty bit from a SHSTK PTE to cause a page fault upon the next SHSTK access. At that time, fix the PTE and copy/re-use the page. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 15 +++++++++++++++ include/asm-generic/pgtable.h | 8 ++++++++ mm/memory.c | 7 ++++++- 3 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 59274e2c1ac4..75dddc3d8451 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -887,3 +887,18 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_mkdirty_shstk(pte); + else + return pte; +} + +inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) +{ + return (vm_flags & VM_SHSTK); +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 359fb935ded6..30ac390fb2d4 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1143,4 +1143,12 @@ static inline bool arch_has_pfn_modify_check(void) #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +#define pte_set_vma_features(pte, vma) pte +#define arch_copy_pte_mapping(vma_flags) false +#else +pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +bool arch_copy_pte_mapping(vm_flags_t vm_flags); +#endif + #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/memory.c b/mm/memory.c index 4ad2d293ddc2..f6b2e1ece4ab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -775,7 +775,8 @@ copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * If it's a COW mapping, write protect it both * in the parent and the child */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { + if ((is_cow_mapping(vm_flags) && pte_write(pte)) || + arch_copy_pte_mapping(vm_flags)) { ptep_set_wrprotect(src_mm, addr, src_pte); pte = pte_wrprotect(pte); } @@ -2218,6 +2219,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2291,6 +2293,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = mk_pte(new_page, vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -2801,6 +2804,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte = mk_pte(page, vma->vm_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); + pte = pte_set_vma_features(pte, vma); vmf->flags &= ~FAULT_FLAG_WRITE; ret |= VM_FAULT_WRITE; exclusive = RMAP_EXCLUSIVE; @@ -2943,6 +2947,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry = mk_pte(page, vma->vm_page_prot); if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); + entry = pte_set_vma_features(entry, vma); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); From patchwork Mon Nov 19 21:47:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689545 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F90213BB for ; Mon, 19 Nov 2018 21:55:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7199C2A583 for ; Mon, 19 Nov 2018 21:55:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 627A12A6EE; Mon, 19 Nov 2018 21:55:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D363C2A583 for ; Mon, 19 Nov 2018 21:55:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F31116B1C9C; Mon, 19 Nov 2018 16:54:28 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EB9FB6B1C9D; Mon, 19 Nov 2018 16:54:28 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C966B6B1C9E; Mon, 19 Nov 2018 16:54:28 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 827AD6B1C9C for ; Mon, 19 Nov 2018 16:54:28 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id g63-v6so27543532pfc.9 for ; Mon, 19 Nov 2018 13:54:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=HLd5GusG71aEYLdIrdzVRcaP/S9spbOy4f1Vn6OuBlU=; b=P5UbOMrT1ElP7aXNZyi2CxS2A6IxcbEZ9uXDRG9C1JwXf0pqz6/5rJKKQqI+0PhunG LNykTBX9ZBPOmbupP8xiS147jKCoE9ju+9hZTaUnOhKf7TmzF0ks2OHY+UPFCihVFIYs 5o3vcDGgB26UdhgTIHiyXjYMCCLBJPZp0SgsOlIqokKetGTcABz3CZOOkDN3/Rtf9OMK /DKKq/TQVsYNJjg8HbR3aikVfzuOyGWQsWTbuzqQAstJ5M3nOIOwZCJ5AGtzbcKiS4Ia efjy5lq1rJ+bTvhHcQecIVMtFSMiobhpwr4KzKu5UlzevJGQB6/GqNV5s42LDBm30J7I bZaw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJL5EICAAT5NwB17a0lEzrb1JQCankNUon++HcOl8t5L36jlnUn eqLSI78ee42fVP/BmN9g2iHjheAwaIhi/cA/Fz60C3o4COZy4SKw26aibnsRXIIlk1QW2OP7fhe wKeEog0Yj9OZlVZexAi+I/PMCQRptsklJIzqiMSHQHNgK18nvaB05Z5y8KHFXn+D/vQ== X-Received: by 2002:a63:1c09:: with SMTP id c9mr21370237pgc.200.1542664468215; Mon, 19 Nov 2018 13:54:28 -0800 (PST) X-Google-Smtp-Source: AJdET5faftZYGEAGBkTx/eiMvIGSLXnHdb3Kl9vJNiXvJcfPHuvkJ/qMjYjH7KXD8tTFAYa2zceR X-Received: by 2002:a63:1c09:: with SMTP id c9mr21370213pgc.200.1542664467441; Mon, 19 Nov 2018 13:54:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664467; cv=none; d=google.com; s=arc-20160816; b=GzkoYOe7LVBKBIdMO6lgcWTLNPOmxcDkNUFt6on+8ixprpfUFIBJqe/DnX988PtByE 4exE85ATDV7p46iw1Mmhkd61uWjWBOnPO7i5/avDLU4i5cLVeM52iYtDMPXM4OGZLVPk LtO2KJFoWJ6x3lKr/a87YlHn+wyxR+9zFGzCpqTf/4M37vEtOZVJlZuZ3dvF35D4AaWc YKAR6KHdRPMIQakrccxSFC4Q9WyfItXZBbI2DlkOApbbKa6eFID1oilvYj9Ohh4gjQhb +9hHEeJqSSLGI/XRkDRjMfPcvESm0pfQXjW3TwWgGN7hxwybXWPrAcBp24IeFbOqyfXz zATg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=HLd5GusG71aEYLdIrdzVRcaP/S9spbOy4f1Vn6OuBlU=; b=EotLB0QcvPxNlHme590gE0jtTO27TgiZNTbF3cD0Yfbwl+je1X2pO8tuplLXZmDPH6 6BF9H6xPywgHrkbRdQ2Vn9LFbypgncpOC96/4kwiCBNZg/bG5q7xAff/JXkJAsHl5qIE JkRBiC/wbDKc1O0k1cPxn6DNAEC7DACfgIlr281cpYTSTL6uRBmVWPPe8DqypcDlObAT 8V1ivdVIYqlYohV6qx+ivkhanNhNuJzzgLwY+ZvSpPi0/26hnV+Mb4eXLi2mDrcYe+af uZVGdhxCrsaDCCpdAnRg8JulW7iRtlRZRbBzf0IE9sOsUhJXXzJrdXtXGhnb84Z3QHo3 wkvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id j7si4446716plb.91.2018.11.19.13.54.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:27 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319889" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:26 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 16/26] mm: Handle THP/HugeTLB shadow stack page fault Date: Mon, 19 Nov 2018 13:47:59 -0800 Message-Id: <20181119214809.6086-17-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch implements THP shadow stack (SHSTK) copying in the same way as in the previous patch for regular PTE. In copy_huge_pmd(), clear the dirty bit from the PMD to cause a page fault upon the next SHSTK access to the PMD. At that time, fix the PMD and copy/re-use the page. Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 2 ++ mm/huge_memory.c | 4 ++++ 3 files changed, 14 insertions(+) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 75dddc3d8451..4275c80f5832 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -897,6 +897,14 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) return pte; } +inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_mkdirty_shstk(pmd); + else + return pmd; +} + inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 30ac390fb2d4..b0b375d8bb34 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1145,9 +1145,11 @@ static inline bool arch_has_pfn_modify_check(void) #ifndef CONFIG_ARCH_HAS_SHSTK #define pte_set_vma_features(pte, vma) pte +#define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); +pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); #endif diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 55478ab3c83b..12148a5b60e0 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -597,6 +597,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry = mk_huge_pmd(page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); page_add_new_anon_rmap(page, vma, haddr, true); mem_cgroup_commit_charge(page, memcg, false, true); lru_cache_add_active_or_unevictable(page, vma); @@ -1209,6 +1210,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pte_t entry; entry = mk_pte(pages[i], vma->vm_page_prot); entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = pte_set_vma_features(entry, vma); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -1293,6 +1295,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = pmd_mkyoung(orig_pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1)) update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); ret |= VM_FAULT_WRITE; @@ -1365,6 +1368,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) pmd_t entry; entry = mk_huge_pmd(new_page, vma->vm_page_prot); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); + entry = pmd_set_vma_features(entry, vma); pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd); page_add_new_anon_rmap(new_page, vma, haddr, true); mem_cgroup_commit_charge(new_page, memcg, false, true); From patchwork Mon Nov 19 21:48:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADCBE6C5 for ; Mon, 19 Nov 2018 21:55:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C8A62A583 for ; Mon, 19 Nov 2018 21:55:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CF2D2A6EE; Mon, 19 Nov 2018 21:55:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0D142A583 for ; Mon, 19 Nov 2018 21:55:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6C5A6B1C9D; Mon, 19 Nov 2018 16:54:29 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF89A6B1C9E; Mon, 19 Nov 2018 16:54:29 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C95B56B1C9F; Mon, 19 Nov 2018 16:54:29 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 7C1866B1C9D for ; Mon, 19 Nov 2018 16:54:29 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id i22-v6so26683967pfj.1 for ; Mon, 19 Nov 2018 13:54:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=62OUP1tZ91xCDgL5KWdzAo5LEZnvgw3ubKR50PUY7os=; b=uavV20F0uqb9CUFGOMliI+N/MDxsOzBpJdK4oG+EOwJP4bxN7vnkWFc+eVUir19Wch jxpglHVxlCpK2SrNG+7OifXNNH/pxF7HbwPGpX/QfujhBrxoMtHg00K6N3uhsWDO6mf/ p3X+BdJ3IWswnMEEJXO1pFST7kti8azKP+NbcatukrUsW3oMw57HDQ/Xm1nCd2Yfenb1 GB42TS4+tGu6gND95LhpUBgfD4h6YcMFoq8JBwOu10poUl3MrsiXn39IvQkCq4cMwAUq +krLL1PjU9ktYPy0aEQI9PdI5kpA3QIdVW2F6M5nGs8t2xqsfVXOQwZI3a+wIRyxPWqY S1sw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJEFthYEctiBNs7i9Grlac2Ji1YrYspgqoKr/X9YyAMbwwyzFZm XzgYV/lDw+ImEqT8KiSfqwM8B9Em4JCGv/7zp8+ezN3zIQB3I81pQCurJ7WbTi9dGHqvTvP6U0C AIfR0c1IGrmFj8wUcbLtvQhxBUKXaZyM3XT27KH7tcBg2/7TpzBlTBpwGrURaduUyMw== X-Received: by 2002:a63:9e0a:: with SMTP id s10mr21849648pgd.239.1542664469156; Mon, 19 Nov 2018 13:54:29 -0800 (PST) X-Google-Smtp-Source: AJdET5cLTWDxW4RQj0BAzB9S7Mi84fArwZRFlGDXlDU71/D5xsxqtGLpd28GSd0X6oSMGldamc14 X-Received: by 2002:a63:9e0a:: with SMTP id s10mr21849612pgd.239.1542664468354; Mon, 19 Nov 2018 13:54:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664468; cv=none; d=google.com; s=arc-20160816; b=WkG4DDajYyiNEL+lJiPbdpJMs76OeK0fIR9VDm7/rbHf7ansiBBJUu4/4s0oxi9bke 0fw4ztMJrtjOwDIK6ow+x5FhlMz12rZ06EPCvl6Wm44X4LFZQKe8TrxHZM6CMcvIA0MQ SmYx8NdwDC5gx6aw2ykGy/Ga6zuHngImDdV2+45/Ug4nkwv0rOJv1+Tr/GUxRfu/UOc6 MPGSbkjdz3sqOhB0IQJGrFVuHfwxhQZ6c5XgZ8kAuJUlqv6XeTw0KilnuXJeiPVBkftV rFPU1BXEKZWrJ+TxZBN1nPaYcG0ntzfDlEdrZoo9Ni3apU6MMfuwJmQkjBlfrt+Z5Ih+ 7wbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=62OUP1tZ91xCDgL5KWdzAo5LEZnvgw3ubKR50PUY7os=; b=ztmGhrX48cVRiD0d7CZye9oxLabQ4cL5Dxz+XL9klha1IvcB90l/V5Y43D3Y1nDENg S6uRcnxMJpyWmJypOdzcNlrHHKeo949+XI/8Pnts9zL4/b86xUb9gqrr9oJ21iY9bFgK lVbVUrLStWPzgO6vYVBGYsbDnHFe9LAOV4pRdhYSB3TwW7teKr8FYMVuffJvRb1lvZFk 7QiCHpVg9Z2Y8f3X4VFo+/ImrW8nHW1yJU2Bfrx4+vMx42KXVNmaZgu2gHSYbYgbd9RM r7f/aQaobbxmYGUsZX6iab57e8YNUIqpqdkaVURnSH80dAxg5ECb/elh2aZExVRQ7YWH 8mJw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id g12-v6si41569459pla.351.2018.11.19.13.54.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:28 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:27 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319892" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:26 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 17/26] mm: Update can_follow_write_pte/pmd for shadow stack Date: Mon, 19 Nov 2018 13:48:00 -0800 Message-Id: <20181119214809.6086-18-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP can_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW. A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW. Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive. Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?" Signed-off-by: Yu-cheng Yu --- arch/x86/mm/pgtable.c | 18 ++++++++++++++++++ include/asm-generic/pgtable.h | 4 ++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 32 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 4275c80f5832..7629156d30b0 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -909,4 +909,22 @@ inline bool arch_copy_pte_mapping(vm_flags_t vm_flags) { return (vm_flags & VM_SHSTK); } + +inline bool pte_exclusive(pte_t pte, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pte_dirty_hw(pte); + else + return pte_dirty(pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma) +{ + if (vma->vm_flags & VM_SHSTK) + return pmd_dirty_hw(pmd); + else + return pmd_dirty(pmd); +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */ diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index b0b375d8bb34..c8685df71521 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1147,10 +1147,14 @@ static inline bool arch_has_pfn_modify_check(void) #define pte_set_vma_features(pte, vma) pte #define pmd_set_vma_features(pmd, vma) pmd #define arch_copy_pte_mapping(vma_flags) false +#define pte_exclusive(pte, vma) pte_dirty(pte) +#define pmd_exclusive(pmd, vma) pmd_dirty(pmd) #else pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma); pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma); bool arch_copy_pte_mapping(vm_flags_t vm_flags); +bool pte_exclusive(pte_t pte, struct vm_area_struct *vma); +bool pmd_exclusive(pmd_t pmd, struct vm_area_struct *vma); #endif #endif /* _ASM_GENERIC_PGTABLE_H */ diff --git a/mm/gup.c b/mm/gup.c index aa43620a3270..abc200ed1e4a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -69,10 +69,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, * FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte(pte_t pte, unsigned int flags) +static inline bool can_follow_write(pte_t pte, unsigned int flags, + struct vm_area_struct *vma) { return pte_write(pte) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pte_dirty(pte)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pte_exclusive(pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma, @@ -110,7 +112,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, } if ((flags & FOLL_NUMA) && pte_protnone(pte)) goto no_page; - if ((flags & FOLL_WRITE) && !can_follow_write_pte(pte, flags)) { + if ((flags & FOLL_WRITE) && !can_follow_write(pte, flags, vma)) { pte_unmap_unlock(ptep, ptl); return NULL; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 12148a5b60e0..f7476eeed83a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1403,10 +1403,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) * FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd(pmd_t pmd, unsigned int flags) +static inline bool can_follow_write(pmd_t pmd, unsigned int flags, + struct vm_area_struct *vma) { return pmd_write(pmd) || - ((flags & FOLL_FORCE) && (flags & FOLL_COW) && pmd_dirty(pmd)); + ((flags & FOLL_FORCE) && (flags & FOLL_COW) && + pmd_exclusive(pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, @@ -1419,7 +1421,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, assert_spin_locked(pmd_lockptr(mm, pmd)); - if (flags & FOLL_WRITE && !can_follow_write_pmd(*pmd, flags)) + if (flags & FOLL_WRITE && !can_follow_write(*pmd, flags, vma)) goto out; /* Avoid dumping huge zero page */ From patchwork Mon Nov 19 21:48:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69EBE6C5 for ; Mon, 19 Nov 2018 21:55:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D4852A583 for ; Mon, 19 Nov 2018 21:55:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 512632A6EE; Mon, 19 Nov 2018 21:55:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E19642A583 for ; Mon, 19 Nov 2018 21:55:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80B4F6B1C9E; Mon, 19 Nov 2018 16:54:30 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 765786B1C9F; Mon, 19 Nov 2018 16:54:30 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51C8B6B1CA0; Mon, 19 Nov 2018 16:54:30 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 02D096B1C9F for ; Mon, 19 Nov 2018 16:54:30 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id g24-v6so27483208pfi.23 for ; Mon, 19 Nov 2018 13:54:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Wx+Td2bZIEI5qvhmOGUHF8l1VQAnWxqhL8toaU9RoMc=; b=ql4JDBq2CFNctshCqQ30tn2oJf2iy2WLG/kAuy6qrOwfGf1FimhawlIACjm8z8gXzD xSqToB1NwYSGSPuhSNFb+PC/pCFCXnuFBb466PyWZZ3WExPmlCUzJE5ieFGdzpeskhVp +MGYyKHS3nXojYoxn+FW7KsWMhYnDDiLIYaRsV4+rCyj/eDgT//nmJtrL4t5U6vytBON tL2yuQqqN3nrkvGl2hyZ//OHqQwnmA76wmT4ujYgGMHa5OnrgaWBHiSEDDBwpuC2qRFW QxsqGvG7AB9BG9lBJ3y99Zlsb0K8z5fhYsv1fgHRN3Hg0TuUHYdzv/YJVzANiOSS46Qa NWBQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gK2YPFtNJtdLiAc+dM1eDPtiNkzpMivt/uw7W0+wP2iBJpM7Jpd jvlT+z3mmNqC4JK/csax7dTVZUs8Ob4iFA558nPisLrCP4JEUGhQoHLF+u8GjObI8See6bMQ2A+ 04acmAejYqlKRiL4Y3dhBTVnS8J7W3NPK+/3EJpcd+d6SepNER6Og3DuUdXi8Nu/arQ== X-Received: by 2002:a17:902:780c:: with SMTP id p12mr23692812pll.197.1542664469702; Mon, 19 Nov 2018 13:54:29 -0800 (PST) X-Google-Smtp-Source: AJdET5eVaXpZnXQsctt21k9bZNseKGFDdoZr8xkZTdNrZbcnds7IJDnteITY9kKWz7oX/igWLXdF X-Received: by 2002:a17:902:780c:: with SMTP id p12mr23692789pll.197.1542664469059; Mon, 19 Nov 2018 13:54:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664469; cv=none; d=google.com; s=arc-20160816; b=vPsLwN9tlw1AM6Oi7laB48YSq7X0w5HHSzuAmG84CKH2O6UNTnOeyJmlvfHhvqAfe7 AqTbvrjHL6MOgGbMpHqUq3mcvbz2RubX//HE653LPwqnWNSanfuWa3R63dVfLFspLFJI BccfDO1pyG1mUCkF/hbQsMPkWi/Qd6jbSl2og7xcfG2wS+8vKv/VSslpHSiLaEn6h8JA RP//myGnVMVhfqFohVCNx/sM5cdeNawkLeh6FZLbDZJIrNZULgIk8LWG+RONU0RnNBqr NUwjyZfVxTjZTBTEpwBJaC7kxQ+Hqll6r/1fSfsg2HjMZEjb5bPuyESKKt2rmME1YhdF YHSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Wx+Td2bZIEI5qvhmOGUHF8l1VQAnWxqhL8toaU9RoMc=; b=xRWuO7IRJJv+UtHNixICxfsxH3eAC668axcmXK5W9v6Xn3KNd/xhnZCy0AFzjyicMD cUWO45XsP7ZlOhNMLdJu/GwhBrODzwwh5sjndEZxFJqDhtnD/XRzWN6jOPC/0hQ9lmse 6b4q5S66ey36chw0heJvRMDa+xCR2RZ0UjkZnJky2cSgQyaX1IdFJZtGyn8mi0EPASVP 0ObqTnKKNsFaFv2Wv9l5J9huA234hc7iDjyplZUwEmV35QJdTEiUQOmH6wz2EHeN73uB MnYWwqCijhG2sfSDGI1mnf0Q+O5/P4IIxIGMvI4xMmLiFpSB+/XxKf+SVeUFP1BI5luI m7FA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id g12-v6si41569459pla.351.2018.11.19.13.54.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:29 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319895" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:27 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 18/26] mm: Introduce do_mmap_locked() Date: Mon, 19 Nov 2018 13:48:01 -0800 Message-Id: <20181119214809.6086-19-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that. Signed-off-by: Yu-cheng Yu --- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7873ac3635a7..36f72c4441cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2348,6 +2348,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len) static inline void mm_populate(unsigned long addr, unsigned long len) {} #endif +static inline unsigned long do_mmap_locked(unsigned long addr, + unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags) +{ + struct mm_struct *mm = current->mm; + unsigned long populate; + + down_write(&mm->mmap_sem); + addr = do_mmap(NULL, addr, len, prot, flags, vm_flags, 0, + &populate, NULL); + up_write(&mm->mmap_sem); + + if (populate) + mm_populate(addr, populate); + + return addr; +} + /* These take the mm semaphore themselves */ extern int __must_check vm_brk(unsigned long, unsigned long); extern int __must_check vm_brk_flags(unsigned long, unsigned long, unsigned long); From patchwork Mon Nov 19 21:48:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689553 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5BDEC13BB for ; Mon, 19 Nov 2018 21:55:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D6862A583 for ; Mon, 19 Nov 2018 21:55:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 414C92A6EE; Mon, 19 Nov 2018 21:55:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5324B2A583 for ; Mon, 19 Nov 2018 21:55:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC9966B1CA1; Mon, 19 Nov 2018 16:54:32 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9FE646B1CA2; Mon, 19 Nov 2018 16:54:32 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8500D6B1CA3; Mon, 19 Nov 2018 16:54:32 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 383096B1CA1 for ; Mon, 19 Nov 2018 16:54:32 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id r16so21625966pgr.15 for ; Mon, 19 Nov 2018 13:54:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=P15/ug9tOQuhGhfvIsfPs843GkLiVRcf1x9tfFK4zjY=; b=Dj2GbeafcJpzGwhgyoq1HuN8hhxREffcv/4Jd1J0tNrq/8a7PU3dvQV3xf+xUFsE4F /BSPUXcv4Wy3A3YO6Pue3lfYiMWdx+55KjnoEzMNIAGT3TJ5PvCYjdT2ZSp4MxILx9vr CC5YiMsgSxjkjJiksfo26X1XqlrUKhnecvWss8vwIGfwIta+hrHlbQ4wVRBczTr5Yw1B Wed9BhZOIOnKET3mSNmH2jNbZRy2rNuLq+hxYZGnYVJjQ4Rqnw/JY7sVXOzKgYV75s/k NHQ2Po+81Ww8tzLwT6Lzm8CscSOMxXQ/xpntujmNzQBhaJP0JN8h6Zm/jSgnfKv60AU/ CY7Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gI9Ps+bPW1WIaESdiNMBX0+I0nieV07f+GdU6iZrOucdvhVQi4Y 3ZC6pCVNepxQDOXER0gHP/PPcUFJG+le2hptg0Tda6tKfgG1wB1tTuKGqvUyvDoPdlL1B9ys0Bk eHtMzWbKCvXg4jw/YOAIL22+K+QHQSPs/iDE6FvRCl345OLiOhoRsN7ixtIi66S93dA== X-Received: by 2002:a65:66ce:: with SMTP id c14mr21491397pgw.450.1542664471858; Mon, 19 Nov 2018 13:54:31 -0800 (PST) X-Google-Smtp-Source: AJdET5fNuB9M0om34QBHtbA3Soqd6Ms3RERm/8MtEfk0MUpKTQ4T3JxrcWs7Qag1eqsodHVKRP3s X-Received: by 2002:a65:66ce:: with SMTP id c14mr21491333pgw.450.1542664470118; Mon, 19 Nov 2018 13:54:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664470; cv=none; d=google.com; s=arc-20160816; b=BCfR/YVCF3lns/+wM+RnKnHeLIIau6HMDRv5GFfHJXZVIluGQ7vfYQ3nouCkFMlPqj ukZOhS4PyA+zgcExsfAp/x7hPsYRfsbNp8YcPolfANCi3ln+gFgwXG3zOqYS9Mm8GovG DnkCeSV8IWJGWOfNw/rJ3m8d/wa2bqYM9NbE5qrrSpSyGzQ6UjlGWjmxwxFaL8+MsKYk kb5UEinkTbQXUyWtVtNNoRNONsu4qKv+FjbNmALqvwrZTIcFSzRn3l1LDUN07KnUV5DZ wUSmuo5nxmLq3UEQRwBB/AqNj197/CHkKt2Q8ZMyGbHNzIae4N9wCwKoH/jnXMpxaLnz UDhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=P15/ug9tOQuhGhfvIsfPs843GkLiVRcf1x9tfFK4zjY=; b=EQvM/e9o/dtcpAmIaTmvL/DuZWQOBRqgVZOYpKBC7lWQSr/zuM8+M4jYhoVTdyEcqV sb+Wh/ufs1k7DSY/phsHNU8jknjSRwhWkNQ/oxFw+Cdmn/MXeT4T4AX6F7N6w4A6dqT4 04BPiTpfRLI5nbxnE0R/Ada1izRYfXPB9pJh/SnQLhyE/PjiFpSlraT8vbTq37xliTF3 hG7tg/RXSph9xUaLOuUV42rVtlQmJCDQhJy4jyZOKN92c8LQkvcaqY3jqGcX00kVLv7a aG8kGjHXMzcC8WSeBB779qW8WzQxEBPfsUZn4PGHVyuYpvGI78cLiHohcxz8NrGklTf9 2E/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:30 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319899" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:28 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 19/26] x86/cet/shstk: User-mode shadow stack support Date: Mon, 19 Nov 2018 13:48:02 -0800 Message-Id: <20181119214809.6086-20-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. It has a fixed size of RLIMIT_STACK. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 34 ++++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 15 +++ arch/x86/include/asm/processor.h | 5 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 109 ++++++++++++++++++ arch/x86/kernel/cpu/common.c | 25 ++++ arch/x86/kernel/process.c | 2 + .../arch/x86/include/asm/disabled-features.h | 8 +- 9 files changed, 206 insertions(+), 2 deletions(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h new file mode 100644 index 000000000000..c952a2ec65fe --- /dev/null +++ b/arch/x86/include/asm/cet.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include + +struct task_struct; +/* + * Per-thread CET status + */ +struct cet_status { + unsigned long shstk_base; + unsigned long shstk_size; + unsigned int shstk_enabled:1; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk(void); +void cet_disable_shstk(void); +void cet_disable_free_shstk(struct task_struct *p); +#else +static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline void cet_disable_shstk(void) {} +static inline void cet_disable_free_shstk(struct task_struct *p) {} +#endif + +#define cpu_x86_cet_enabled() \ + (cpu_feature_enabled(X86_FEATURE_SHSTK) || \ + cpu_feature_enabled(X86_FEATURE_IBT)) + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 33833d1909af..3624a11e5ba6 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -75,7 +81,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 80f4a4f38c79..298721ff00f4 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -778,4 +778,19 @@ #define MSR_VM_IGNNE 0xc0010115 #define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0 /* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2 /* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4 /* kernel shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7 /* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8 /* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL +#define MSR_IA32_CET_BITMAP_MASK 0xfffffffffffff000ULL + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 071b2a6fff85..1222dc204c40 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -24,6 +24,7 @@ struct vm86; #include #include #include +#include #include #include @@ -500,6 +501,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET + struct cet_status cet; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 8824d01c0c35..fbb2d91fb756 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c new file mode 100644 index 000000000000..e6726e78e6cd --- /dev/null +++ b/arch/x86/kernel/cet.c @@ -0,0 +1,109 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * cet.c - Control-flow Enforcement (CET) + * + * Copyright (c) 2018, Intel Corporation. + * Yu-cheng Yu + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static int set_shstk_ptr(unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + return 0; +} + +static unsigned long get_shstk_addr(void) +{ + unsigned long ptr; + + if (!current->thread.cet.shstk_enabled) + return 0; + + rdmsrl(MSR_IA32_PL3_SSP, ptr); + return ptr; +} + +int cet_setup_shstk(void) +{ + unsigned long addr, size; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return -EOPNOTSUPP; + + size = rlimit(RLIMIT_STACK); + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + /* + * Return actual error from do_mmap(). + */ + if (addr >= TASK_SIZE_MAX) + return addr; + + set_shstk_ptr(addr + size - sizeof(u64)); + current->thread.cet.shstk_base = addr; + current->thread.cet.shstk_size = size; + current->thread.cet.shstk_enabled = 1; + return 0; +} + +void cet_disable_shstk(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK)) + return; + + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_SHSTK_EN); + wrmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, 0); + current->thread.cet.shstk_enabled = 0; +} + +void cet_disable_free_shstk(struct task_struct *tsk) +{ + if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || + !tsk->thread.cet.shstk_enabled) + return; + + if (tsk == current) + cet_disable_shstk(); + + /* + * Free only when tsk is current or shares mm + * with current but has its own shstk. + */ + if (tsk->mm && (tsk->mm == current->mm) && + (tsk->thread.cet.shstk_base)) { + vm_munmap(tsk->thread.cet.shstk_base, + tsk->thread.cet.shstk_size); + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + } + + tsk->thread.cet.shstk_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index ffb181f959d2..795e195bf2fe 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -52,6 +52,7 @@ #include #include #include +#include #ifdef CONFIG_X86_LOCAL_APIC #include @@ -411,6 +412,29 @@ static __init int setup_disable_pku(char *arg) __setup("nopku", setup_disable_pku); #endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ + if (cpu_x86_cet_enabled()) + cr4_set_bits(X86_CR4_CET); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk(char *s) +{ + /* require an exact match without trailing characters */ + if (s[0] != '\0') + return 0; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_SHSTK); + pr_info("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n"); + return 1; +} +__setup("no_cet_shstk", setup_disable_shstk); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization @@ -1379,6 +1403,7 @@ static void identify_cpu(struct cpuinfo_x86 *c) x86_init_rdrand(c); x86_init_cache_qos(c); setup_pku(c); + setup_cet(c); /* * Clear/Set all flags overridden by options, need do it diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c93fcfdf1673..4a776da4c28c 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -39,6 +39,7 @@ #include #include #include +#include /* * per-CPU TSS segments. Threads are completely 'soft' on Linux, @@ -134,6 +135,7 @@ void flush_thread(void) flush_ptrace_hw_breakpoint(tsk); memset(tsk->thread.tls_array, 0, sizeof(tsk->thread.tls_array)); + cet_disable_shstk(); fpu__clear(&tsk->thread.fpu); } diff --git a/tools/arch/x86/include/asm/disabled-features.h b/tools/arch/x86/include/asm/disabled-features.h index 33833d1909af..3624a11e5ba6 100644 --- a/tools/arch/x86/include/asm/disabled-features.h +++ b/tools/arch/x86/include/asm/disabled-features.h @@ -56,6 +56,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -75,7 +81,7 @@ #define DISABLED_MASK13 0 #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) From patchwork Mon Nov 19 21:48:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DB9C13BB for ; Mon, 19 Nov 2018 21:55:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 100F12A583 for ; Mon, 19 Nov 2018 21:55:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 03F0A2A6F3; Mon, 19 Nov 2018 21:55:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 901BF2A583 for ; Mon, 19 Nov 2018 21:55:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F82C6B1C9F; Mon, 19 Nov 2018 16:54:32 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 484376B1CA3; Mon, 19 Nov 2018 16:54:32 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E5B36B1CA2; Mon, 19 Nov 2018 16:54:32 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id CB3CD6B1C9F for ; Mon, 19 Nov 2018 16:54:31 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id 68so21655045pfr.6 for ; Mon, 19 Nov 2018 13:54:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=E0G/ZhvDtzadXfew8Pq+HzUDsd3S9LSVTFg1o3vV4DM=; b=qxYkD6ZMfapVr6vb+UWL4PtibYiT8LR2p2pe43DR/lZzehmDkmxwvJcJMct+TXm7LD Xwub+FqMzPnxtxuhge4fyP0ToqWEHAbPyiFuBiSn9ezejd4FveolHszUoJ8mEnAHXkT0 BkXzBMvIV1A2L9UZYUf3MpLu9TvacQGEueRCS9HwphV35s9x/q8bS/nzt2D/sKDyGT4/ vc7s8avqzxxZPY/vLnXuIg0h/dYsGgUYQV4gHv1z/h0bUUzMIXVRcPxk0N9sHd/Ie6RJ Gs02flX+0+ePv3BGFAmRJAUO3sPaJOpe7KaP6eeVWFAaSsYSupw3iwvQsyufthMvPXUU pyZw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gLVSZHE9f992RL7Fs5Ti7Q4vPS3MMQSNqFTExnE5N1MkRGkOuMW 7gHscnFnZSG1cTUbHvyyKzbd2/Vg6ABKEPA94/gkInp5FyjOxoWSLvGmkXLvPfdYzZM5ZOFn9aE yOHzOoPCVwRHBYKvadopl+giyaOpWAypPCzQBxsNaXC40A7GSW+lcEeyEGTw/xtWgYw== X-Received: by 2002:a62:5f05:: with SMTP id t5-v6mr24709867pfb.223.1542664471510; Mon, 19 Nov 2018 13:54:31 -0800 (PST) X-Google-Smtp-Source: AJdET5eKWfXZCheBmz8VxJHyK+JsQi7OSLThdA68K0x0dFEY9aXkphWev/ssAv6tQ6TEzGRo9lOG X-Received: by 2002:a62:5f05:: with SMTP id t5-v6mr24709834pfb.223.1542664470773; Mon, 19 Nov 2018 13:54:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664470; cv=none; d=google.com; s=arc-20160816; b=nHbTy4y1tvEX2ZNzEpi3kseawj/fhJ0NOTgc2FOWb55Sc9wh2lj+tjL+BN49OIYOjR WjvCBtXBUOKPzQKMXAwDKagzFz658KBmmePXHEGRWKNdpFUuJE827MY4PIRA9/EW1uaa AJ5+tKvjSpt9pg6Y0dyMW2QO+T75ygyDETRP0RmYuU0W8ChT7GaAZ41roDgruG6LSiv4 BHudgF28u+OGtZuzvCT0cT3XCjdr07Ozqn9tRhpjefjchwNOFptVXUtIIx3SFRJhqfLy vLMam9RYRbyvGYAs17UtaIcdJFjcJ2ziyDr6caGnA/kXirUAD9AbHF9Mi9w+YZvC7A68 IR5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=E0G/ZhvDtzadXfew8Pq+HzUDsd3S9LSVTFg1o3vV4DM=; b=U37Ja338yoyRoL8Cf+oMlA8Se8VhWsqcnSL4EW/WqvJdMu4KPfhBOknJRX6r2HI1wY bJlD6DAUmiL7qWi0FGRpuhll2UGa9h4vgY+Dk25nPB518Q7pOSZf1RgdV1AsLTE+ZkSE 5BgLAvqOBcOxCVQk7ARylL+zcmhX+z2Is8YrCx8mAd8EIMfW+tB8y+7nbgFBwuStN/UD uaD2fBtGQYH1k4ok8cW+xC4ZBfrQ2m7n8HCwpvJ6FtaMGAT60kKMwdII302Pro5Dzkes B6yvhLPU6iIditqgXSPFf73LFgV95TqimkBULPNf+OINi9ELdkjenVChHBivbcWRV64A e3/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:30 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:30 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319902" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:29 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 20/26] x86/cet/shstk: Introduce WRUSS instruction Date: Mon, 19 Nov 2018 13:48:03 -0800 Message-Id: <20181119214809.6086-21-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler. This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does a fixup. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/special_insns.h | 32 ++++++++++++++++++++++++++++ arch/x86/mm/fault.c | 9 ++++++++ 2 files changed, 41 insertions(+) diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 43c029cdc3fe..43957f197a9a 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -237,6 +237,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET +#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + asm_volatile_goto("1: wrussd %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#else +static inline int write_user_shstk_32(unsigned long addr, unsigned int val) +{ + WARN_ONCE(1, "%s used but not supported.\n", __func__); + return -EFAULT; +} +#endif + +static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{ + asm_volatile_goto("1: wrussq %1, (%0)\n" + _ASM_EXTABLE(1b, %l[fail]) + :: "r" (addr), "r" (val) + :: fail); + return 0; +fail: + return -EPERM; +} +#endif /* CONFIG_X86_INTEL_CET */ + #define nop() asm volatile ("nop") diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index c3368fed706c..7b5de629748e 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1318,6 +1318,15 @@ void do_user_addr_fault(struct pt_regs *regs, } flags |= FAULT_FLAG_USER; } else { + /* + * WRUSS is a kernel instruction and but writes + * to user shadow stack. When a fault occurs, + * both X86_PF_USER and X86_PF_SHSTK are set. + * Clear X86_PF_USER from sw_error_code. + */ + if ((hw_error_code & (X86_PF_USER | X86_PF_SHSTK)) == + (X86_PF_USER | X86_PF_SHSTK)) + sw_error_code &= ~X86_PF_USER; if (regs->flags & X86_EFLAGS_IF) local_irq_enable(); } From patchwork Mon Nov 19 21:48:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689555 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 991FF6C5 for ; Mon, 19 Nov 2018 21:55:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87C132A583 for ; Mon, 19 Nov 2018 21:55:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7BB042A6EE; Mon, 19 Nov 2018 21:55:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7FEB22A583 for ; Mon, 19 Nov 2018 21:55:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14ED66B1CA3; Mon, 19 Nov 2018 16:54:34 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0382A6B1CA4; Mon, 19 Nov 2018 16:54:33 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E45126B1CA5; Mon, 19 Nov 2018 16:54:33 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 833A16B1CA3 for ; Mon, 19 Nov 2018 16:54:33 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id w7-v6so24593181plp.9 for ; Mon, 19 Nov 2018 13:54:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0M8t9qnF9rdGhUz2Q9Ni21ToxqMLf0bkIU3r+Pkb4M4=; b=jzm09SDJhQwtGs0hwX/78GrguRmcBmo3wO4RFpltxO898ipFktiKTBntsuMCMtS0SG Ip5hJnE8OfBzCtfbn+Nx9AB/cCrnfuWHLQJl3nLQIbEQnQbEQvMo7RChCPqJzLHtL2UG lV0ePA8sxbt33HGLyxAh5DFu7vRTz4cmkUzPgqmwmzTe77z46b1vIQC6nhxrKsi4aSFk C2N0wdfJZzfeKOytucc22S4qt93prGXREFjJKsgrtJxRE3eg9ilrS3RPgKy65m9JALyb EjWcS79R5UNhFq6Zsw1V+//OYCb21BHlFe/1RH9NT2uinmljEdGBUBZa0tsTwE5pxaBh oCog== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJMCRtCK8GbpGBbjjCYOcO6aab67IPjlgtVf6fMit7h2LEaUNHI JhritOEz9E5zjvy8RIwfhGjyPFkiZyzMoKpIuBRw4U1IAiQuwO6BxmJgVE21rk+cQ9SeYThmafa J7RPOnRQAGB83Sn6aB++SOsLL4BfeVEmKijr+KHusT4XwBfhvQltFM0aBp11XEFEEtw== X-Received: by 2002:a63:af18:: with SMTP id w24mr21879128pge.385.1542664473162; Mon, 19 Nov 2018 13:54:33 -0800 (PST) X-Google-Smtp-Source: AJdET5fa30HAWfgBmMg+a4uUL+kCzKraIcpNTy9lxe/N+gjqD1kdv7GSriAJ1ocPGl24jNrvpoEQ X-Received: by 2002:a63:af18:: with SMTP id w24mr21879083pge.385.1542664471822; Mon, 19 Nov 2018 13:54:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664471; cv=none; d=google.com; s=arc-20160816; b=y9/WTSdvS42TL6D6NTtI15Rx4RAOwgHnGv7bxM9z+ZbW6svb0HzQvo/anyUB/FnOVQ J/nXQ5R2TBsL11orCAhBmP3McJZgtVIiWqbXrb/4S7LUKl6gZPRW09sy5pUnWZgLqT5u 8ZB+axQ2jf/b4pClVhJMls1buN8ABkzfH1Ww5LMnahKh0LtaI6lfCUb4/RQMLkGcr4h2 ajAi9/6tIi8KtJ6yIYqRse2yVl/EiKKL/f8GkQPNvESqwwBrevEHTVOPGk9b5gUg45cR IB1F4Xjk4AXguY512AMpb8q2AFUDjV9RNaP7SDgf54QeepK9P5uEiJt5UMM02M0y7rqu dmzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0M8t9qnF9rdGhUz2Q9Ni21ToxqMLf0bkIU3r+Pkb4M4=; b=QrDjd2MPvzciLjup2C8XVBJ9irmp1mgkRazlK91O8vTs3pIHHdeM95zpL15yyBju3c z5TnJ+C3MYA3c0ysGFPvjR72aPfs0C5xgy42Ujdpr1UdkCkFyC6gijUacEz8J7QvCyA1 a5cWf5G1wC/TgZzgngen5RwUXNUuJPsBE1SytZ9wdYdSkQCUxm5VG8arJ7AVvZSsS+Vc o2snndp89IAETNOk+7f+c4fFqfxXR4Rz2EZZ4sA/h2wdvxzUOL4rdN+6jox17SrFBRsI Qw8pFtxylWT4gpZzuR5El1J751L2/UywxzjBbXixCgPb4VoUjuh6TCn+ci9Vgf6W6k/v QVgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:31 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319905" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:30 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 21/26] x86/cet/shstk: Signal handling for shadow stack Date: Mon, 19 Nov 2018 13:48:04 -0800 Message-Id: <20181119214809.6086-22-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer. Signed-off-by: Yu-cheng Yu --- arch/x86/ia32/ia32_signal.c | 21 +++++ arch/x86/include/asm/cet.h | 5 + arch/x86/include/asm/sighandling.h | 5 + arch/x86/include/uapi/asm/sigcontext.h | 15 +++ arch/x86/kernel/cet.c | 126 +++++++++++++++++++++++++ arch/x86/kernel/signal.c | 97 +++++++++++++++++++ 6 files changed, 269 insertions(+) diff --git a/arch/x86/ia32/ia32_signal.c b/arch/x86/ia32/ia32_signal.c index 86b1341cba9a..fd4d18ab82f9 100644 --- a/arch/x86/ia32/ia32_signal.c +++ b/arch/x86/ia32/ia32_signal.c @@ -34,6 +34,7 @@ #include #include #include +#include /* * Do a signal return; undo the signal stack. @@ -108,6 +109,9 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, 1); + if (!err) + err = restore_sigcontext_ext(buf); + force_iret(); return err; @@ -209,6 +213,17 @@ static int ia32_setup_sigcontext(struct sigcontext_32 __user *sc, return err; } +static unsigned long alloc_sigcontext_ext(unsigned long sp) +{ + /* + * sigcontext_ext is at: fpu + fpu_user_xstate_size + + * FP_XSTATE_MAGIC2_SIZE, then aligned to 8. + */ + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + return sp; +} + /* * Determine which stack to use.. */ @@ -234,6 +249,7 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, if (fpu->initialized) { unsigned long fx_aligned, math_size; + sp = alloc_sigcontext_ext(sp); sp = fpu__alloc_mathframe(sp, 1, &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned, @@ -277,6 +293,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, if (ia32_setup_sigcontext(&frame->sc, fpstate, regs, set->sig[0])) return -EFAULT; + if (setup_sigcontext_ext(ksig, fpstate)) + return -EFAULT; if (_COMPAT_NSIG_WORDS > 1) { if (__copy_to_user(frame->extramask, &set->sig[1], @@ -384,6 +402,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index c952a2ec65fe..3af544aed800 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -19,10 +19,15 @@ struct cet_status { int cet_setup_shstk(void); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); +int cet_restore_signal(unsigned long ssp); +int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} +static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } +static inline int cet_setup_signal(bool ia32, unsigned long rstor, + unsigned long *new_ssp) { return -EINVAL; } #endif #define cpu_x86_cet_enabled() \ diff --git a/arch/x86/include/asm/sighandling.h b/arch/x86/include/asm/sighandling.h index bd26834724e5..23014b4082de 100644 --- a/arch/x86/include/asm/sighandling.h +++ b/arch/x86/include/asm/sighandling.h @@ -17,4 +17,9 @@ void signal_fault(struct pt_regs *regs, void __user *frame, char *where); int setup_sigcontext(struct sigcontext __user *sc, void __user *fpstate, struct pt_regs *regs, unsigned long mask); +#ifdef CONFIG_X86_64 +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fpu); +int restore_sigcontext_ext(void __user *fpu); +#endif + #endif /* _ASM_X86_SIGHANDLING_H */ diff --git a/arch/x86/include/uapi/asm/sigcontext.h b/arch/x86/include/uapi/asm/sigcontext.h index 844d60eb1882..e3b08d1c0d3b 100644 --- a/arch/x86/include/uapi/asm/sigcontext.h +++ b/arch/x86/include/uapi/asm/sigcontext.h @@ -196,6 +196,21 @@ struct _xstate { /* New processor state extensions go here: */ }; +/* + * Sigcontext extension (struct sc_ext) is located after + * sigcontext->fpstate. Because currently only the shadow + * stack pointer is saved there and the shadow stack depends + * on XSAVES, we can find sc_ext from sigcontext->fpstate. + * + * The 64-bit fpstate has a size of fpu_user_xstate_size, plus + * FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext + * is located at the end of sigcontext->fpstate, aligned to 8. + */ +struct sc_ext { + unsigned long total_size; + unsigned long ssp; +}; + /* * The 32-bit signal frame: */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index e6726e78e6cd..44904c90d347 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -18,6 +18,7 @@ #include #include #include +#include static int set_shstk_ptr(unsigned long addr) { @@ -46,6 +47,80 @@ static unsigned long get_shstk_addr(void) return ptr; } +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64(token) ((token & TOKEN_MODE_MASK) == TOKEN_MODE_64) +#define IS_TOKEN_32(token) ((token & TOKEN_MODE_MASK) == 0) + +/* + * Verify the restore token at the address of 'ssp' is + * valid and then set shadow stack pointer according to the + * token. + */ +static int verify_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long token; + + *new_ssp = 0; + + if (!IS_ALIGNED(ssp, 8)) + return -EINVAL; + + if (get_user(token, (unsigned long __user *)ssp)) + return -EFAULT; + + /* Is 64-bit mode flag correct? */ + if (ia32 && !IS_TOKEN_32(token)) + return -EINVAL; + else if (!IS_TOKEN_64(token)) + return -EINVAL; + + token &= ~TOKEN_MODE_MASK; + + /* + * Restore address properly aligned? + */ + if ((!ia32 && !IS_ALIGNED(token, 8)) || !IS_ALIGNED(token, 4)) + return -EINVAL; + + /* + * Token was placed properly? + */ + if ((ALIGN_DOWN(token, 8) - 8) != ssp) + return -EINVAL; + + *new_ssp = token; + return 0; +} + +/* + * Create a restore token on the shadow stack. + * A token is always 8-byte and aligned to 8. + */ +static int create_rstor_token(bool ia32, unsigned long ssp, + unsigned long *new_ssp) +{ + unsigned long addr; + + *new_ssp = 0; + + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4)) + return -EINVAL; + + addr = ALIGN_DOWN(ssp, 8) - 8; + + /* Is the token for 64-bit? */ + if (!ia32) + ssp |= TOKEN_MODE_64; + + if (write_user_shstk_64(addr, ssp)) + return -EFAULT; + + *new_ssp = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; @@ -107,3 +182,54 @@ void cet_disable_free_shstk(struct task_struct *tsk) tsk->thread.cet.shstk_enabled = 0; } + +int cet_restore_signal(unsigned long ssp) +{ + unsigned long new_ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + err = verify_rstor_token(in_ia32_syscall(), ssp, &new_ssp); + + if (err) + return err; + + return set_shstk_ptr(new_ssp); +} + +/* + * Setup the shadow stack for the signal handler: first, + * create a restore token to keep track of the current ssp, + * and then the return address of the signal handler. + */ +int cet_setup_signal(bool ia32, unsigned long rstor_addr, + unsigned long *new_ssp) +{ + unsigned long ssp; + int err; + + if (!current->thread.cet.shstk_enabled) + return 0; + + ssp = get_shstk_addr(); + err = create_rstor_token(ia32, ssp, new_ssp); + + if (err) + return err; + + if (ia32) { + ssp = *new_ssp - sizeof(u32); + err = write_user_shstk_32(ssp, (unsigned int)rstor_addr); + } else { + ssp = *new_ssp - sizeof(u64); + err = write_user_shstk_64(ssp, rstor_addr); + } + + if (err) + return err; + + set_shstk_ptr(ssp); + return 0; +} diff --git a/arch/x86/kernel/signal.c b/arch/x86/kernel/signal.c index 92a3b312a53c..72b70b0c1c49 100644 --- a/arch/x86/kernel/signal.c +++ b/arch/x86/kernel/signal.c @@ -46,6 +46,7 @@ #include #include +#include #define COPY(x) do { \ get_user_ex(regs->x, &sc->x); \ @@ -152,6 +153,10 @@ static int restore_sigcontext(struct pt_regs *regs, err |= fpu__restore_sig(buf, IS_ENABLED(CONFIG_X86_32)); +#ifdef CONFIG_X86_64 + err |= restore_sigcontext_ext(buf); +#endif + force_iret(); return err; @@ -237,6 +242,17 @@ static unsigned long align_sigframe(unsigned long sp) return sp; } +static unsigned long alloc_sigcontext_ext(unsigned long sp) +{ + /* + * sigcontext_ext is at: fpu + fpu_user_xstate_size + + * FP_XSTATE_MAGIC2_SIZE, then aligned to 8. + */ + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) + sp -= (sizeof(struct sc_ext) + 8); + return sp; +} + static void __user * get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, void __user **fpstate) @@ -266,6 +282,7 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size, } if (fpu->initialized) { + sp = alloc_sigcontext_ext(sp); sp = fpu__alloc_mathframe(sp, IS_ENABLED(CONFIG_X86_32), &buf_fx, &math_size); *fpstate = (void __user *)sp; @@ -493,6 +510,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig, err |= setup_sigcontext(&frame->uc.uc_mcontext, fp, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fp); + if (err) return -EFAULT; @@ -576,6 +596,9 @@ static int x32_setup_rt_frame(struct ksignal *ksig, regs, set->sig[0]); err |= __copy_to_user(&frame->uc.uc_sigmask, set, sizeof(*set)); + if (!err) + err = setup_sigcontext_ext(ksig, fpstate); + if (err) return -EFAULT; @@ -707,6 +730,80 @@ setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) } } +#ifdef CONFIG_X86_64 +static int copy_ext_from_user(struct sc_ext *ext, void __user *fpu) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_from_user(ext, p, sizeof(*ext))) + return -EFAULT; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + return 0; +} + +static int copy_ext_to_user(void __user *fpu, struct sc_ext *ext) +{ + void __user *p; + + if (!fpu) + return -EINVAL; + + if (ext->total_size != sizeof(*ext)) + return -EINVAL; + + p = fpu + fpu_user_xstate_size + FP_XSTATE_MAGIC2_SIZE; + p = (void __user *)ALIGN((unsigned long)p, 8); + + if (copy_to_user(p, ext, sizeof(*ext))) + return -EFAULT; + + return 0; +} + +int restore_sigcontext_ext(void __user *fp) +{ + int err = 0; + + if (cpu_feature_enabled(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + + err = copy_ext_from_user(&ext, fp); + + if (!err) + err = cet_restore_signal(ext.ssp); + } + + return err; +} + +int setup_sigcontext_ext(struct ksignal *ksig, void __user *fp) +{ + int err = 0; + + if (cpu_feature_enabled(X86_FEATURE_SHSTK) && fp) { + struct sc_ext ext = {0, 0}; + unsigned long rstor; + + rstor = (unsigned long)ksig->ka.sa.sa_restorer; + err = cet_setup_signal(is_ia32_frame(ksig), rstor, &ext.ssp); + if (!err) { + ext.total_size = sizeof(ext); + err = copy_ext_to_user(fp, &ext); + } + } + + return err; +} +#endif + static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { From patchwork Mon Nov 19 21:48:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689557 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48C5913BB for ; Mon, 19 Nov 2018 21:55:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39E172A583 for ; Mon, 19 Nov 2018 21:55:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2DC782A6EE; Mon, 19 Nov 2018 21:55:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 314B62A583 for ; Mon, 19 Nov 2018 21:55:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 284A36B1CA5; Mon, 19 Nov 2018 16:54:35 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 22D566B1CA7; Mon, 19 Nov 2018 16:54:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F05CF6B1CA6; Mon, 19 Nov 2018 16:54:34 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id A0FBD6B1CA4 for ; Mon, 19 Nov 2018 16:54:34 -0500 (EST) Received: by mail-pg1-f200.google.com with SMTP id 143so18932131pgc.3 for ; Mon, 19 Nov 2018 13:54:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=i+o+Tte0Hohgkol2C0NE2FsUdfd0oWCemKTt+n0WcGM=; b=VqsRWRnL41SFYrc4LhlTAyABDa8yuQ86oa4orsTOpC1AzUM73NwT0cgSfaVSKA4uq+ Scyjd8HrrW6WktxEzRtkp8c9jRIs/L+FObT1nJNDv7uQzo6oGDEZNaMV6Z88SaSgP4u1 UW5mu4gbxUzlhJDacOdjZvq+R33vSHxcXnktBeV0Wb49ohzrcVkeJGqPE5Yvi9AddeaK Sdd/vrBklEnD4FlX2BsxhE/Wa7iFRu7a3C5nJkb7MsTi6JxVNAoo+Pg1qF4ylNgu74Yx xB2LkQ/xNyogPNz2PIAe8lPoAVBpAO786xR+FZw5ADsvf2l2HJrlFUCNcAjYaW+Amdlj dzDw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gLk5vtMgHDeUxyGc6+h8vEVHu8mf7kyhn6ksccU3O7hjsAPGWZD uDb8hI5n3ySn4sv0zS/6yyITM2Rn1VIt2ssceYZJtxb52FX7ksZ4wwvlFOrKhyZpVRvlxi5Qxdm VDyJuPgYUaJVJUhDr8cUjNZttfADVTUMHMXSTfZa8clDd694EJzIshSYestnVH6NiaQ== X-Received: by 2002:a17:902:110a:: with SMTP id d10-v6mr24411683pla.85.1542664474297; Mon, 19 Nov 2018 13:54:34 -0800 (PST) X-Google-Smtp-Source: AJdET5cwMBLfAhckIknDHdVIY9PWgBMKMRbZPLBZEKAJhVmcs1NE65NdiWeTcxu8MJzv7T9W/w/S X-Received: by 2002:a17:902:110a:: with SMTP id d10-v6mr24411633pla.85.1542664472681; Mon, 19 Nov 2018 13:54:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664472; cv=none; d=google.com; s=arc-20160816; b=R+YKUlXDjbjMe7q9i0WP59kKDuUlhl89rSEMInu4VHmeXAcr6SePUna1omrDA/h8w2 FClP2mgl57DGYOacHxMIX/oJ5c4EsRfOC74V+7CfaywcvcKZmCi0yojNtLWeUZjn8A3C rgIQYRxOOzJrP+uKi24pGXo9RI1OT4abTmr/AtRABZybH8AeKKXn032Oy3Iq5aLFzH3j NdiRoVQhc5X8Uh5XUTkwDLzJ3hGaR/uvFWJvBZIM9AVqbIWDpXxgXVpEfTDJZWWoZ6Zs kD3OoaS3p9UrBpY4lzd7pvtARKmMs5kPHFT5zHN9tX7hEQqvqqdthgnLUNJlO1tFwpbk 6GZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=i+o+Tte0Hohgkol2C0NE2FsUdfd0oWCemKTt+n0WcGM=; b=gz8y5rMr4Z18qtkzoAkBMf0324xKQYdQoO9DLF5c2NYUVH18/gHwrl6Ys3gqUFT7NO mD6vPXiWyIMoQsiAWnEjJEH8qCjc06BndCLU/1cZUS8QddUsYmEeOWoAroHZC+FemGkT 2Nc+uxNKaNSyl97bB+n1USLsebyk4L6ZW89h0vqDnZShir0mev1x5kThpGRkaQILEajz Rrntokpt7tFmbYxJEv3fGp2vNHLjsaPpiZPADkgaD+ppSnxYFjNr/e8Au9jJKRA0nWOQ ew5daue8JSgyVrNPjhQgkieNaTC8pA6a03bdjHacfL5YMv1yK7fXqOvuJSFTqP59LrYu ta7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:32 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319908" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:31 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 22/26] x86/cet/shstk: ELF header parsing of Shadow Stack Date: Mon, 19 Nov 2018 13:48:05 -0800 Message-Id: <20181119214809.6086-23-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 4 + arch/x86/include/asm/elf.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/elf.c | 358 +++++++++++++++++++++++ fs/binfmt_elf.c | 15 + include/uapi/linux/elf.h | 1 + 7 files changed, 400 insertions(+) create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/elf.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 86fb68f496a6..73dfb94cde71 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1925,6 +1925,9 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_PROGRAM_PROPERTIES + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n @@ -1932,6 +1935,7 @@ config X86_INTEL_SHADOW_STACK_USER select ARCH_USES_HIGH_VMA_FLAGS select X86_INTEL_CET select ARCH_HAS_SHSTK + select ARCH_HAS_PROGRAM_PROPERTIES ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h index 69c0f892e310..557ed0ba71c7 100644 --- a/arch/x86/include/asm/elf.h +++ b/arch/x86/include/asm/elf.h @@ -381,4 +381,9 @@ struct va_alignment { extern struct va_alignment va_align; extern unsigned long align_vdso_addr(unsigned long); + +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +extern int arch_setup_features(void *ehdr, void *phdr, struct file *file, + bool interp); +#endif #endif /* _ASM_X86_ELF_H */ diff --git a/arch/x86/include/uapi/asm/elf_property.h b/arch/x86/include/uapi/asm/elf_property.h new file mode 100644 index 000000000000..af361207718c --- /dev/null +++ b/arch/x86/include/uapi/asm/elf_property.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _UAPI_ASM_X86_ELF_PROPERTY_H +#define _UAPI_ASM_X86_ELF_PROPERTY_H + +/* + * pr_type + */ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* + * Bits for GNU_PROPERTY_X86_FEATURE_1_AND + */ +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + +#endif /* _UAPI_ASM_X86_ELF_PROPERTY_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index fbb2d91fb756..36b14ef410c8 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y) diff --git a/arch/x86/kernel/elf.c b/arch/x86/kernel/elf.c new file mode 100644 index 000000000000..60e396e2abe9 --- /dev/null +++ b/arch/x86/kernel/elf.c @@ -0,0 +1,358 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Look at an ELF file's .note.gnu.property and determine if the file + * supports shadow stack and/or indirect branch tracking. + * The path from the ELF header to the note section is the following: + * elfhdr->elf_phdr->elf_note->property[]. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * The .note.gnu.property layout: + * + * struct elf_note { + * u32 n_namesz; --> sizeof(n_name[]); always (4) + * u32 n_ndescsz;--> sizeof(property[]) + * u32 n_type; --> always NT_GNU_PROPERTY_TYPE_0 + * }; + * char n_name[4]; --> always 'GNU\0' + * + * struct { + * struct property_x86 { + * u32 pr_type; + * u32 pr_datasz; + * }; + * u8 pr_data[pr_datasz]; + * }[]; + */ + +#define BUF_SIZE (PAGE_SIZE / 4) + +struct property_x86 { + u32 pr_type; + u32 pr_datasz; +}; + +typedef bool (test_item_fn)(void *buf, u32 *arg); +typedef void *(next_item_fn)(void *buf, u32 *arg); + +static inline bool test_note_type_0(void *buf, u32 *align) +{ + struct elf_note *n = buf; + + return ((n->n_type == NT_GNU_PROPERTY_TYPE_0) && (n->n_namesz == 4) && + (memcmp(n + 1, "GNU", 4) == 0)); +} + +static inline void *next_note(void *buf, u32 *align) +{ + struct elf_note *n = buf; + u64 size; + + if (check_add_overflow((u64)sizeof(*n), (u64)n->n_namesz, &size)) + return NULL; + + size = round_up(size, *align); + + if (check_add_overflow(size, (u64)n->n_descsz, &size)) + return NULL; + + size = round_up(size, *align); + + if (buf + size < buf) + return NULL; + else + return (buf + size); +} + +static inline bool test_property_x86(void *buf, u32 *max_type) +{ + struct property_x86 *pr = buf; + + /* + * Property types must be in ascending order. + * Keep track of the max when testing each. + */ + if (pr->pr_type > *max_type) + *max_type = pr->pr_type; + + return (pr->pr_type == GNU_PROPERTY_X86_FEATURE_1_AND); +} + +static inline void *next_property(void *buf, u32 *max_type) +{ + struct property_x86 *pr = buf; + + if ((buf + sizeof(*pr) + pr->pr_datasz < buf) || + (pr->pr_type > GNU_PROPERTY_X86_FEATURE_1_AND) || + (pr->pr_type > *max_type)) + return NULL; + else + return (buf + sizeof(*pr) + pr->pr_datasz); +} + +/* + * Scan 'buf' for a pattern; return true if found. + * *pos is the distance from the beginning of buf to where + * the searched item or the next item is located. + */ +static int scan(u8 *buf, u32 buf_size, int item_size, test_item_fn test_item, + next_item_fn next_item, u32 *arg, u32 *pos) +{ + int found = 0; + u8 *p, *max; + + max = buf + buf_size; + if (max < buf) + return 0; + + p = buf; + + while ((p + item_size < max) && (p + item_size > buf)) { + if (test_item(p, arg)) { + found = 1; + break; + } + + p = next_item(p, arg); + } + + *pos = (p + item_size <= buf) ? 0 : (u32)(p - buf); + return found; +} + +/* + * Search a NT_GNU_PROPERTY_TYPE_0 for GNU_PROPERTY_X86_FEATURE_1_AND. + */ +static int find_feature_x86(struct file *file, unsigned long desc_size, + loff_t file_offset, u8 *buf, u32 *feature) +{ + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + u32 last_pr = 0; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < desc_size; done += buf_pos) { + read_size = desc_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) + return (ret < 0) ? ret : -EIO; + + ret = 0; + found = scan(buf, read_size, sizeof(struct property_x86), + test_property_x86, next_property, + &last_pr, &buf_pos); + + if ((!buf_pos) || found) + break; + + file_offset += buf_pos - read_size; + } + + if (found) { + struct property_x86 *pr = + (struct property_x86 *)(buf + buf_pos); + + if (pr->pr_datasz == 4) { + u32 *max = (u32 *)(buf + read_size); + u32 *data = (u32 *)((u8 *)pr + sizeof(*pr)); + + if (data + 1 <= max) { + *feature = *data; + } else { + file_offset += buf_pos - read_size; + file_offset += sizeof(*pr); + ret = kernel_read(file, feature, 4, + &file_offset); + } + } + } + + return ret; +} + +/* + * Search a PT_NOTE segment for NT_GNU_PROPERTY_TYPE_0. + */ +static int find_note_type_0(struct file *file, unsigned long note_size, + loff_t file_offset, u32 align, u32 *feature) +{ + u8 *buf; + u32 buf_pos; + unsigned long read_size; + unsigned long done; + int found = 0; + int ret = 0; + + buf = kmalloc(BUF_SIZE, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + *feature = 0; + buf_pos = 0; + + for (done = 0; done < note_size; done += buf_pos) { + read_size = note_size - done; + if (read_size > BUF_SIZE) + read_size = BUF_SIZE; + + ret = kernel_read(file, buf, read_size, &file_offset); + + if (ret != read_size) { + ret = (ret < 0) ? ret : -EIO; + kfree(buf); + return ret; + } + + /* + * item_size = sizeof(struct elf_note) + elf_note.n_namesz. + * n_namesz is 4 for the note type we look for. + */ + ret = scan(buf, read_size, sizeof(struct elf_note) + 4, + test_note_type_0, next_note, + &align, &buf_pos); + + file_offset += buf_pos - read_size; + + if (ret && !found) { + struct elf_note *n = + (struct elf_note *)(buf + buf_pos); + u64 start = round_up(sizeof(*n) + n->n_namesz, align); + u64 total = 0; + + if (check_add_overflow(start, (u64)n->n_descsz, &total)) { + ret = -EINVAL; + break; + } + total = round_up(total, align); + + ret = find_feature_x86(file, n->n_descsz, + file_offset + start, + buf, feature); + found++; + file_offset += total; + buf_pos += total; + } else if (!buf_pos || ret) { + ret = 0; + *feature = 0; + break; + } + } + + kfree(buf); + return ret; +} + +#ifdef CONFIG_COMPAT +static int check_notes_32(struct file *file, struct elf32_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 4)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +#ifdef CONFIG_X86_64 +static int check_notes_64(struct file *file, struct elf64_phdr *phdr, + int phnum, u32 *feature) +{ + int i; + int err = 0; + + for (i = 0; i < phnum; i++, phdr++) { + if ((phdr->p_type != PT_NOTE) || (phdr->p_align != 8)) + continue; + + err = find_note_type_0(file, phdr->p_filesz, phdr->p_offset, + phdr->p_align, feature); + if (err) + return err; + } + + return 0; +} +#endif + +int arch_setup_features(void *ehdr_p, void *phdr_p, + struct file *file, bool interp) +{ + int err = 0; + u32 feature = 0; + + struct elf64_hdr *ehdr64 = ehdr_p; + + if (!cpu_x86_cet_enabled()) + return 0; + + if (ehdr64->e_ident[EI_CLASS] == ELFCLASS64) { + struct elf64_phdr *phdr64 = phdr_p; + + err = check_notes_64(file, phdr64, ehdr64->e_phnum, + &feature); + if (err < 0) + goto out; + } else { +#ifdef CONFIG_COMPAT + struct elf32_hdr *ehdr32 = ehdr_p; + + if (ehdr32->e_ident[EI_CLASS] == ELFCLASS32) { + struct elf32_phdr *phdr32 = phdr_p; + + err = check_notes_32(file, phdr32, ehdr32->e_phnum, + &feature); + if (err < 0) + goto out; + } +#else + WARN_ONCE(1, "Exec of 32-bit app, but CONFIG_COMPAT is not enabled.\n"); + return -ENOTSUPP; +#endif + } + + memset(¤t->thread.cet, 0, sizeof(struct cet_status)); + + if (cpu_feature_enabled(X86_FEATURE_SHSTK)) { + if (feature & GNU_PROPERTY_X86_FEATURE_1_SHSTK) { + err = cet_setup_shstk(); + if (err < 0) + goto out; + } + } + +out: + return err; +} diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 54207327f98f..007ff0fbae84 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1081,6 +1081,21 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES + if (interpreter) { + retval = arch_setup_features(&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + } else { + retval = arch_setup_features(&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + + if (retval < 0) + goto out_free_dentry; +#endif + if (elf_interpreter) { unsigned long interp_map_addr = 0; diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index c5358e0ae7c5..5ef25a565e88 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -372,6 +372,7 @@ typedef struct elf64_shdr { #define NT_PRFPREG 2 #define NT_PRPSINFO 3 #define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5 #define NT_AUXV 6 /* * Note to userspace developers: size of NT_SIGINFO note may increase From patchwork Mon Nov 19 21:48:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689559 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6A3613BB for ; Mon, 19 Nov 2018 21:55:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A746F2A583 for ; Mon, 19 Nov 2018 21:55:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9AD9E2A6EE; Mon, 19 Nov 2018 21:55:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00B2B2A583 for ; Mon, 19 Nov 2018 21:55:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89F996B1CA4; Mon, 19 Nov 2018 16:54:35 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 829186B1CA6; Mon, 19 Nov 2018 16:54:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 606F36B1CA7; Mon, 19 Nov 2018 16:54:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 154FB6B1CA4 for ; Mon, 19 Nov 2018 16:54:35 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id d11-v6so24533306plo.17 for ; Mon, 19 Nov 2018 13:54:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/HKdGiQvtW5jf7nkNgP3KN8Y7Fo/pfx9me0h5Ldadis=; b=uYD2PiVmdW/+yzWVjipwGuyvMYLUf9S0KytLmX/30+wfh0PxdtLcB1DSWKMpMjOP30 VPsAU5TN6pcswILAlb8xmP51uDo1D3OEuRIVqc14rwqKTml0hC+m/aYQLAONX0+xKxPT Ie1pFxQxeD/m+rrmGv7t0IosVqDkHroAo6Pu0hTasXVgZvQK6iKi3sXFNn142dUH2v5p 1B7xOVmwoOJsOI4N5vnVvjLTXff3lVqb5f7yDM4DxalWcoFDMSkPni2qmNsitUuL33Fg 6RbVJygLNbMqzh2KSeUfBIzB9QrlCLDbn8RoH6HkO6dYzMq9il0hD1sD/UqkpmmaH8jR U2oA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gKun7HH3PzsHR3mylt/l/fz3H15UWLCGAy1zErYPTipxsVmdcSr LiXOlQiOQS7zD3XAo/H8N5HRJvPfZkZYTVLVXmgfDiKTnL8/arHv+/YWhI19KlU1C4CDsnu01L3 IZa90nG7tZsNOMsqHL5BhQ0PDoV7KugmzJqUY8JddrQbgbz66eN5Bx9hq73g9LnrYyA== X-Received: by 2002:a62:b15:: with SMTP id t21mr11895390pfi.136.1542664474743; Mon, 19 Nov 2018 13:54:34 -0800 (PST) X-Google-Smtp-Source: AJdET5cUwF5cvFm1gdNpa1hgDndNv94kJUwQNZGqrsPPNFNpzfhtnakhhgRZXrB1XyQjpaNfDoJm X-Received: by 2002:a62:b15:: with SMTP id t21mr11895344pfi.136.1542664473483; Mon, 19 Nov 2018 13:54:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664473; cv=none; d=google.com; s=arc-20160816; b=OsmMpoPxuilba3OTVs+5CQN2+sUqUXNkR8YjQu51PVzcAUi4z+p7rSgaT/JSgL5sHD doeg3EqyzQSzMsAfzrWeAM8wB4f96dJV+albuGu2Vfx1fJj3vTKGtMk7MgG7iYU7lA37 b/K5kCgu2mTgL0diOijcuwPWzj4zyMucxmZl00gtMDbjzUuZQ+Q5MaamC+77HzyRfXk5 9Uk5cSIPkZEl3r698tR2fKiyEeWwMfzp2QLQg4+ysYtockESyOZVaJk3ETkojgTI5p+T bX//UKxEeaoHscLZM+VoFGYKwIgTN6QsXUtGLvsbW2K1fb1ETZgT+AEh+5MoB/es/Jjk Gcig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/HKdGiQvtW5jf7nkNgP3KN8Y7Fo/pfx9me0h5Ldadis=; b=MJyJUvH1vLb4SmTGoj7PSD/0r0LorpshC3F62JsG1Vmadu4b4bePV8ICyfHKqtCzYa kCTaLuUKI9UJbSeWcf3YJW69MzEiMfZ3lxZ8osZ/wrphISmkw7qzOZniaB20XywwFEQO NAvEn/pXomPpJP4J1vPc0Tq0axuXENgQPzmmNHgjosjlTX++4jEmP5bK2zCLHwRmnl8+ SRscX9OV3WO27Vg9ZHQbvjQ1FkVB2L/aVpZxL2lkn2w1Zi9c1Od/o1SZ7WSZLFUhUSqO b5pb8DuJjn17azA5nwyUVxmWsxv48zwsWaoDkI3E8yxdHwrjnTPJvlJaVn7M2HKQUSCE qkZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:33 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319912" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:32 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 23/26] x86/cet/shstk: Handle thread shadow stack Date: Mon, 19 Nov 2018 13:48:06 -0800 Message-Id: <20181119214809.6086-24-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The shadow stack for clone/fork is handled as the following: (1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child. It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK. (2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK. (3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret. This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches. A 64-bit SHSTK has a fixed size of RLIMIT_STACK. A compat-mode thread SHSTK has a fixed size of 1/4 RLIMIT_STACK. This allows more threads to share a 32-bit address space. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 40 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 ++++++ 5 files changed, 53 insertions(+) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 3af544aed800..5957e7257d83 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -17,12 +17,14 @@ struct cet_status { #ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk(void); +int cet_setup_thread_shstk(struct task_struct *p); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else static inline int cet_setup_shstk(void) { return -EINVAL; } +static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 0ca50611e8ce..57c1f6c42bef 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -223,6 +224,8 @@ do { \ #else #define deactivate_mm(tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 44904c90d347..0e3e7a2c6f80 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -145,6 +145,46 @@ int cet_setup_shstk(void) return 0; } +int cet_setup_thread_shstk(struct task_struct *tsk) +{ + unsigned long addr, size; + struct cet_user_state *state; + + if (!current->thread.cet.shstk_enabled) + return 0; + + state = get_xsave_addr(&tsk->thread.fpu.state.xsave, + XFEATURE_MASK_SHSTK_USER); + + if (!state) + return -EINVAL; + + size = rlimit(RLIMIT_STACK); + + /* + * Compat-mode pthreads share a limited address space. + * If each function call takes an average of four slots + * stack space, we need 1/4 of stack size for shadow stack. + */ + if (in_compat_syscall()) + size /= 4; + + addr = do_mmap_locked(0, size, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + + if (addr >= TASK_SIZE_MAX) { + tsk->thread.cet.shstk_base = 0; + tsk->thread.cet.shstk_size = 0; + tsk->thread.cet.shstk_enabled = 0; + return -ENOMEM; + } + + state->user_ssp = (u64)(addr + size - sizeof(u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; + return 0; +} + void cet_disable_shstk(void) { u64 r; diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 4a776da4c28c..440f012ef925 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -125,6 +125,7 @@ void exit_thread(struct task_struct *tsk) free_vm86(t); + cet_disable_free_shstk(tsk); fpu__drop(fpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 0e0b4288a4b2..3b371a57426e 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -456,6 +456,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp, if (sp) childregs->sp = sp; + /* Allocate a new shadow stack for pthread */ + if ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM) { + err = cet_setup_thread_shstk(p); + if (err) + goto out; + } + err = -ENOMEM; if (unlikely(test_tsk_thread_flag(me, TIF_IO_BITMAP))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr, From patchwork Mon Nov 19 21:48:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689561 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D6206C5 for ; Mon, 19 Nov 2018 21:55:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 310F02A583 for ; Mon, 19 Nov 2018 21:55:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 24E862A6EE; Mon, 19 Nov 2018 21:55:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC0062A583 for ; Mon, 19 Nov 2018 21:55:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0870E6B1CA6; Mon, 19 Nov 2018 16:54:36 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0151B6B1CA7; Mon, 19 Nov 2018 16:54:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2F4F6B1CA8; Mon, 19 Nov 2018 16:54:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 8F9B66B1CA7 for ; Mon, 19 Nov 2018 16:54:35 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id w7-v6so24593239plp.9 for ; Mon, 19 Nov 2018 13:54:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=u4R5q5qMWOYQdLv8bWwZPIQsE0aBT0lrX51J+drB2sQ=; b=PZ7+Krh1Fkhw6ANUT+aecy+RpEsCkLb09ESw7kAePC60ojCVtns4xpdC63raugF31T 7p7/87xMMP5SIXb4jgOpBd92lnP6iVk8EPWkeWVqzvs2f89/Sa4xpDUO3uERAxcpte22 Pq4g6dxwnaH3QOq3fhh3h928CZj3NusPVQhebjb27usMeNCJaUu1F8YTL/QbPj5FTmU5 jlHgrdK3PeYU/AKaVqHsuyC75jNi3D5m1enJYVekWXZA/7elwM+AZTsIgpwFraCDEuQE DZGORiThR2GuWKD4k7nCnn2NzVd1HDS/o4EZhVpMwD2STiNBbjo6cQaDndpf7QuJbYqT NNXQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gKlapnNRfS9ADfgyiITPwZQ7ISyjW7ARcgLRTdvRHvUgLqkR4b9 bbPW2qr1t/soZ2ZBcIHXGo+qAnMTsGpCwKxMm3LdD57JyEkTKI5c6C65kQiuGhp0Qj/ZKjUqlfS IYnpbVSZWcsf9KrjgAGDS4fRDQL6jnNB+y38oz/OAXt+nMAb0cyAgi1+SzOLbcg+i9A== X-Received: by 2002:a63:36c4:: with SMTP id d187mr21256341pga.404.1542664475278; Mon, 19 Nov 2018 13:54:35 -0800 (PST) X-Google-Smtp-Source: AJdET5eGnuDoN0dX+f9CAQlbOZNpipMZyBxSYDSfXV8bc/WOf2FWapX37zO0afCPk6P/Y6iSguXU X-Received: by 2002:a63:36c4:: with SMTP id d187mr21256301pga.404.1542664474409; Mon, 19 Nov 2018 13:54:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664474; cv=none; d=google.com; s=arc-20160816; b=o4+EUQS4kNqdhTfmvzKMtms55aGImzUrYXvlPze/IfQMnIEjI/MDJmUoJQI8aWHpXU PsA9spcG+LcWwRMpxuqL5r8YWkZxDEvHl0g2mQJZlVfNxe2NAWDMwWoCB58C1BEZv1xD FFCJl4K1coMez1hbJfTN/sgYxLquD39NNkBTiIv84Rxx1XKAsnltq+qSMNnQqQj/Oc8l bHnkiYOmWjt7eBpbNd2Ek+Ow9v2htTIVrrejLQ2++zdx+huyLbcpVPbnn/Y8niKlJt6u NZ4LG3d1qqqreamrL0tx8/h4kfHV0GJ5pCIPx/Fvi+hgGjOZbwZJoer0oLQ4CfywtoTP taZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=u4R5q5qMWOYQdLv8bWwZPIQsE0aBT0lrX51J+drB2sQ=; b=q50z3nsMvejcdWMSZBjzh18ivwLnFqzpo5kEee5vlogd8HK7g4RkGuYFQa1jcPxw9q sFEpmPexg5a5eybJHSV7LkQMqmd7hIu71Rr58uIIraNvOegpa5XnKttuFNGltJHQr/8M FPvaABWbrJ99fF/Zi4sMC6poWUCMv+vG8p5eHgDGHYS4zp0MxoUNEuq8/IXvX68sNvgA f+ydiE8nplawOXuXqcEYw0TDsMFlLTvRq4R4Ofps6BON9DzHMa3tBZLd3P0FjEY0bgFw 1DKUZRCE8p5dP1Yg+PLf+fBq+ZilH84t/2mCNgA1J9KAbn1CeuwFdJhdlRJ1B6GbRtpb ADRg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:34 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319916" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:33 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 24/26] mm/mmap: Add Shadow stack pages to memory accounting Date: Mon, 19 Nov 2018 13:48:07 -0800 Message-Id: <20181119214809.6086-25-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add shadow stack pages to memory accounting. Signed-off-by: Yu-cheng Yu --- mm/mmap.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 30836512ca79..9560d69fa08c 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1701,6 +1701,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) if (file && is_file_hugepages(file)) return 0; + if (arch_copy_pte_mapping(vm_flags)) + return 1; + return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE; } @@ -3319,6 +3322,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; + else if (arch_copy_pte_mapping(flags)) + mm->data_vm += npages; } static vm_fault_t special_mapping_fault(struct vm_fault *vmf); From patchwork Mon Nov 19 21:48:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689563 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A22BF13BB for ; Mon, 19 Nov 2018 21:55:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 901242A583 for ; Mon, 19 Nov 2018 21:55:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 82DFE2A6EE; Mon, 19 Nov 2018 21:55:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C6C982A583 for ; Mon, 19 Nov 2018 21:55:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C0E76B1CA8; Mon, 19 Nov 2018 16:54:37 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 320786B1CA9; Mon, 19 Nov 2018 16:54:37 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C2C76B1CAA; Mon, 19 Nov 2018 16:54:37 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BFDFD6B1CA8 for ; Mon, 19 Nov 2018 16:54:36 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id 143so18932186pgc.3 for ; Mon, 19 Nov 2018 13:54:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uOY1gJNSay0ExN2qHZQckVGn5IyQf4mJzXB6bHUI0KM=; b=RXTOBKZuwd7jyZJHokJEDka3t3oa1I1fUwTyIpVocHKimXe7eEw5VmYRTFaoYQAznM qgSy2s5pEKuQl64iFCGnqEGP+XJfMxOXczgDcmV57CKYryCodLGNMdkQ3HJ0ydBa1PgX 11UgMfQxivyodpAZWlXeCBp0N2BESKc+4NzqztvumFhQl25WPdWh1mzIyCG9eTpn1fu4 cLeaRlVWifeVsrMqzDy+Z5c1wfMblps5TD03GYyU5mVOvSO6k8LYS0hCOvxi9ZPLQrkT CCCin9n/g3jjZycqa4jrN3FhHHjEgg7RWZuHXNr0lsfxCqQXdMdTFYqUHRkvCPVDPzPG cfMA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJbVjABCo8VGYKIe4pCgc/+i1gU3CitY+klo586qa6kdqoe8F3k OFPzlz5S8JFZOG3SRVlT0AE8mIfnqADjcpqQVe5h4xHYHLvtOvJJ2fTMtjUFvKEkz4a2VAaiIsC ctvbFlDnObOG+yAakEFsIQdECyWBWSeikIwoFQHI9hTT+YCW2xEPIaisQnAJYWuKubw== X-Received: by 2002:a63:2d82:: with SMTP id t124mr21571741pgt.260.1542664476428; Mon, 19 Nov 2018 13:54:36 -0800 (PST) X-Google-Smtp-Source: AJdET5ciVvheO8A5pKhO9WsTx09XYU7V+4tfy78sY/3+0E+EkuHvGP0x2En3B7qVITfW25K33fI3 X-Received: by 2002:a63:2d82:: with SMTP id t124mr21571704pgt.260.1542664475283; Mon, 19 Nov 2018 13:54:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664475; cv=none; d=google.com; s=arc-20160816; b=Jb4zNQkdoLyrmd+FkbH5ydaTzNhtSZj4BaDsRG68cTK2AS1m59OZDpGiTQ78Kc30c3 PZsq/pTVmjsCVPPGN1FY1ubz6IG9hR2tOsGbReC0dfXBJNq/O2X8vLIz/CSZwvc/AY3i C8uCXpxyWowDrsTi0T3EkOyDlfr3SqfJY3lFl9JCBkfiuIexKpP7dw5KxNxTA2RTvqWS +oSFxeY5gJ2sjdwWSNPa5h+F9EcmlYucEuhyYag4nvuHBYD2aoR/OymGqg5nAqwV/BUL /IKCHw1QACHZUqSLqVBa6uonU8gvIug0rMTn+iLTOU4VWBbCcbLgDxECkpIVR1Mu3b4/ A6MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uOY1gJNSay0ExN2qHZQckVGn5IyQf4mJzXB6bHUI0KM=; b=SeLB7Emb1eBKDtt299JD4Nq4TQl1eZijdGA9nQA579d7tTBqmz4w7lGV2ROMyMFkI3 BbfkXtILTE45FCKpxgZPc3SU7IyCDuOpS+cuJ8zmFgUm7eOs1RPcdR4Wr8w2CRDOV8Ok ZHQpL/e2QntAxqtaozoWLD/hIHktqv87IQQOyvmWHMAQLfm1qE7W0C8RRCNFAKoUy51y E2QcOL5L1jZA8e+De94aDbWouH8yI1CSegL9BjtoaQvVl709Q8pABXWGi66kpgwQBLkl q//fdjjeJRMNQt3iAt0UgE8T1HBkvT/qQNNhtcobcrYOcTCj1Kk2rqBnsvPZ17BfmfNr g51Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:35 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319922" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:34 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 25/26] x86/cet/shstk: Add arch_prctl functions for Shadow Stack Date: Mon, 19 Nov 2018 13:48:08 -0800 Message-Id: <20181119214809.6086-26-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP arch_prctl(ARCH_X86_CET_STATUS, unsigned long *addr) Return CET feature status. The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information: *addr = SHSTK/IBT status *(addr + 1) = SHSTK base address *(addr + 2) = SHSTK size arch_prctl(ARCH_X86_CET_DISABLE, unsigned long features) Disable CET features specified in 'features'. Return -EPERM if CET is locked. arch_prctl(ARCH_X86_CET_LOCK) Lock in CET feature. arch_prctl(ARCH_X86_CET_ALLOC_SHSTK, unsigned long *addr) Allocate a new SHSTK. The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK. There is no CET enabling arch_prctl function. By design, CET is enabled automatically if the binary and the system can support it. The parameters passed are always unsigned 64-bit. When an ia32 application passing pointers, it should only use the lower 32 bits. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 29 +++++++++++ arch/x86/kernel/cet_prctl.c | 86 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 4 +- 6 files changed, 128 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cet_prctl.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 5957e7257d83..29d88e4d8d5d 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -12,19 +12,24 @@ struct task_struct; struct cet_status { unsigned long shstk_base; unsigned long shstk_size; + unsigned int locked:1; unsigned int shstk_enabled:1; }; #ifdef CONFIG_X86_INTEL_CET +int prctl_cet(int option, unsigned long arg2); int cet_setup_shstk(void); int cet_setup_thread_shstk(struct task_struct *p); +int cet_alloc_shstk(unsigned long *arg); void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(unsigned long ssp); int cet_setup_signal(bool ia32, unsigned long rstor, unsigned long *new_ssp); #else +static inline int prctl_cet(int option, unsigned long arg2) { return -EINVAL; } static inline int cet_setup_shstk(void) { return -EINVAL; } static inline int cet_setup_thread_shstk(struct task_struct *p) { return 0; } +static inline int cet_alloc_shstk(unsigned long *arg) { return -EINVAL; } static inline void cet_disable_shstk(void) {} static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(unsigned long ssp) { return -EINVAL; } diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 5a6aac9fa41f..d962f0ec9ccf 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -14,4 +14,9 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_X86_CET_STATUS 0x3001 +#define ARCH_X86_CET_DISABLE 0x3002 +#define ARCH_X86_CET_LOCK 0x3003 +#define ARCH_X86_CET_ALLOC_SHSTK 0x3004 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 36b14ef410c8..b9e6cdc6b4f7 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -139,7 +139,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 0e3e7a2c6f80..1b5d1ce4df8f 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -121,6 +121,35 @@ static int create_rstor_token(bool ia32, unsigned long ssp, return 0; } +int cet_alloc_shstk(unsigned long *arg) +{ + unsigned long len = *arg; + unsigned long addr; + unsigned long token; + unsigned long ssp; + + addr = do_mmap_locked(0, len, PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, VM_SHSTK); + if (addr >= TASK_SIZE_MAX) + return -ENOMEM; + + /* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + + if (!in_ia32_syscall()) + token |= TOKEN_MODE_64; + ssp -= 8; + + if (write_user_shstk_64(ssp, token)) { + vm_munmap(addr, len); + return -EINVAL; + } + + *arg = addr; + return 0; +} + int cet_setup_shstk(void) { unsigned long addr, size; diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c new file mode 100644 index 000000000000..320dbb620d61 --- /dev/null +++ b/arch/x86/kernel/cet_prctl.c @@ -0,0 +1,86 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* See Documentation/x86/intel_cet.txt. */ + +static int handle_get_status(unsigned long arg2) +{ + unsigned int features = 0; + unsigned long shstk_base, shstk_size; + unsigned long buf[3]; + + if (current->thread.cet.shstk_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + + shstk_base = current->thread.cet.shstk_base; + shstk_size = current->thread.cet.shstk_size; + + buf[0] = (unsigned long)features; + buf[1] = shstk_base; + buf[2] = shstk_size; + return copy_to_user((unsigned long __user *)arg2, buf, + sizeof(buf)); +} + +static int handle_alloc_shstk(unsigned long arg2) +{ + int err = 0; + unsigned long arg; + unsigned long addr = 0; + unsigned long size = 0; + + if (get_user(arg, (unsigned long __user *)arg2)) + return -EFAULT; + + size = arg; + err = cet_alloc_shstk(&arg); + if (err) + return err; + + addr = arg; + if (put_user(addr, (unsigned long __user *)arg2)) { + vm_munmap(addr, size); + return -EFAULT; + } + + return 0; +} + +int prctl_cet(int option, unsigned long arg2) +{ + if (!cpu_x86_cet_enabled()) + return -EINVAL; + + switch (option) { + case ARCH_X86_CET_STATUS: + return handle_get_status(arg2); + + case ARCH_X86_CET_DISABLE: + if (current->thread.cet.locked) + return -EPERM; + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) + cet_disable_free_shstk(current); + + return 0; + + case ARCH_X86_CET_LOCK: + current->thread.cet.locked = 1; + return 0; + + case ARCH_X86_CET_ALLOC_SHSTK: + return handle_alloc_shstk(arg2); + + default: + return -EINVAL; + } +} diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 440f012ef925..f240fce2b20f 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -792,7 +792,7 @@ long do_arch_prctl_common(struct task_struct *task, int option, return get_cpuid_mode(); case ARCH_SET_CPUID: return set_cpuid_mode(task, cpuid_enabled); + default: + return prctl_cet(option, cpuid_enabled); } - - return -EINVAL; } From patchwork Mon Nov 19 21:48:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 10689565 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9946D13BB for ; Mon, 19 Nov 2018 21:55:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A3792A583 for ; Mon, 19 Nov 2018 21:55:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D87F2A6EE; Mon, 19 Nov 2018 21:55:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C65692A583 for ; Mon, 19 Nov 2018 21:55:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEB2D6B1CAA; Mon, 19 Nov 2018 16:54:38 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A71C46B1CAB; Mon, 19 Nov 2018 16:54:38 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84DAE6B1CAC; Mon, 19 Nov 2018 16:54:38 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 384DE6B1CAA for ; Mon, 19 Nov 2018 16:54:38 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id 89so11003925ple.19 for ; Mon, 19 Nov 2018 13:54:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=d+pGDwpHdvzPdHC1q0xwRn+lq7FqvCE4GJTOUMF3PPh59zagmb4HAIv1QeERhmXEyh E8HbLKjFawmQXXjYqh1qQa3joLeqw3r2Xxglu7e1YNKZH4b1dxMY2FCICaNZMdxlc76/ u1GzQi9+uCgnPkRQyJDlu/xgVra8/OhEQF3PU8FPzD8JAJQbTHlClwMQSH7NiiiXISVC VhW2w0PVay+5t3FMXCIaCfMxKtGyXjMvDQ3xDxAAZv068B5lmema9uuZr7gZvT/8wpmB hq9N73Ffgs6rHQjYbAujbhUrfum4rjDiEeAlxmKygnFGesUAFyJ5yJkDnPSQT7Wl+m50 MzRA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gJ5yKYWJaA+mbzGAZeCi8cTnzQ0A+ZUkTu16b3Pn0Ffr5onnWcV PzoRaegoTOPby0E1gx5fNK+S9IFV4SP53zcSgWM6WGBAuCKADihUB2q7re7vSHIhJQVLJJH5Cmy G/FtizL2HU8k1UFtAW99OsHoCnrQG+Mc1J7uYucj7D5jmaWwmd5ADmXFNscFHLBnrew== X-Received: by 2002:a17:902:104:: with SMTP id 4-v6mr24681083plb.184.1542664477879; Mon, 19 Nov 2018 13:54:37 -0800 (PST) X-Google-Smtp-Source: AJdET5e5+TgTdRyc3LfUbYdHVNtDb0mM7LDxt31pWUM7UTQkPL/FgovWMQWEIxXnSW33RsBGhp7y X-Received: by 2002:a17:902:104:: with SMTP id 4-v6mr24681011plb.184.1542664476256; Mon, 19 Nov 2018 13:54:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542664476; cv=none; d=google.com; s=arc-20160816; b=mX5mC7uL2S6ewB//6jNCkntp4m96xw/rcf+SF+KB6MEtaKIhL1KZYoUElhoIK7EKno yyQiKWk/Hq6AXJef/1+bjjaCTf/SeCaLkF/OdZyOWGN79dYTxiAEy1rUKBxmx6eFJFDC Rb5plmMt4lYauAARSFQw8uBN/0k0Zqc8GmfDiaUD6aPH6RzpOzyL8A/qdyYYLcZ3yigi zGSJgNA21ZZlIi4wGd5cTMjS/nkcL1A8bGMlPWPNpk4n7ad9CTj6GnYP7emym03FxOCa 4VUqOGMO8/ibWvrV9kpQgEvhnXCSOwu2HUHtiXR8i/XOw/FIfGyjchrI/mBIuvj8Nwui ZNhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=OQ5kV+Ft3hkRBLg3gYwc7poXPXYMd+Ka9Q+KpVCdFWE=; b=MdGJFWiWJQm/BMl23WCUVF9GbdQavvro56jMjitBaf0v/X8P3E38nioNCF2z3Gs9KA 9hCxQQkLZ614oPMItSaiOHDpUn/B5+mqBtPstVgbiwTKeJaVqHYvOuznAC/W2Obs1C0K P59vGctXn6HTv7zoqw9WyxY3hc8yxqByg++yc2JhLuyXwKxfrUuu5XbaHZJCOxvE+aFd l1VjaPndPwzXMRpWx6wM1UvwhaxZmIRnl6dY0UxO62YuStgjg+27IRovyqVP0A5Z/X2y XRq+8m+5y+XzXQ0gR5aCezK3fnDM+diDW2FcrrrR8ag13EbIBxf8IlwVYUSPvBIjdb8j /7ww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id o11si40652866pgd.234.2018.11.19.13.54.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 19 Nov 2018 13:54:36 -0800 (PST) Received-SPF: pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yu-cheng.yu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=yu-cheng.yu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 13:54:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,254,1539673200"; d="scan'208";a="93319926" Received: from yyu32-desk1.sc.intel.com ([143.183.136.147]) by orsmga008.jf.intel.com with ESMTP; 19 Nov 2018 13:54:34 -0800 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue Cc: Yu-cheng Yu Subject: [RFC PATCH v6 26/26] x86/cet/shstk: Add Shadow Stack instructions to opcode map Date: Mon, 19 Nov 2018 13:48:09 -0800 Message-Id: <20181119214809.6086-27-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181119214809.6086-1-yu-cheng.yu@intel.com> References: <20181119214809.6086-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the following shadow stack management instructions. INCSSP: Increment shadow stack pointer by the steps specified. RDSSP: Read SSP register into a GPR. SAVEPREVSSP: Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack. RSTORSSP: Restore from a "restore token" pointed by a GPR to SSP. WRSS: Write to kernel-mode shadow stack (kernel-mode instruction). WRUSS: Write to user-mode shadow stack (kernel-mode instruction). SETSSBSY: Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR. CLRSSBSY: Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index e0b85930dd77..c5e825d44766 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd @@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff @@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable @@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable