From patchwork Tue Aug 13 20:53:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092895 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 480751399 for ; Tue, 13 Aug 2019 21:03:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39AAB2868D for ; Tue, 13 Aug 2019 21:03:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2CBA02873A; Tue, 13 Aug 2019 21:03:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC1292868D for ; Tue, 13 Aug 2019 21:03:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F61E6B0298; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9AA636B0296; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 623E36B0299; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 3672D6B0296 for ; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id CD8FD181AC9B6 for ; Tue, 13 Aug 2019 21:03:41 +0000 (UTC) X-FDA: 75818631042.07.brick39_203089ac9991e X-HE-Tag: brick39_203089ac9991e X-Filterd-Recvd-Size: 4056 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:41 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275962" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:38 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 01/14] x86/cet/ibt: Add Kconfig option for user-mode Indirect Branch Tracking Date: Tue, 13 Aug 2019 13:53:46 -0700 Message-Id: <20190813205359.12196-2-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The user-mode indirect branch tracking support is done mostly by GCC to insert ENDBR64/ENDBR32 instructions at branch targets. The kernel provides CPUID enumeration and feature setup. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 16 ++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 23 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 7d13ba326962..666dc66a382e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1940,6 +1940,9 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_AS_LIMIT + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n @@ -1961,6 +1964,19 @@ config X86_INTEL_SHADOW_STACK_USER If unsure, say y. +config X86_INTEL_BRANCH_TRACKING_USER + prompt "Intel Indirect Branch Tracking for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select X86_INTEL_CET + select ARCH_HAS_AS_LIMIT + select ARCH_USE_GNU_PROPERTY + ---help--- + Indirect Branch Tracking provides hardware protection against return-/jmp- + oriented programming attacks. + + If unsure, say y + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 0b2e9df48907..25372cc4a303 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -155,6 +155,13 @@ ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER endif endif +# Check compiler ibt support +ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + ifeq ($(call cc-option-yn, -fcf-protection=branch), n) + $(error CONFIG_X86_INTEL_BRANCH_TRACKING_USER not supported by compiler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug From patchwork Tue Aug 13 20:53:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092897 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9718C912 for ; Tue, 13 Aug 2019 21:04:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88BF2286D1 for ; Tue, 13 Aug 2019 21:04:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7C621286BF; Tue, 13 Aug 2019 21:04:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6FAD2871C for ; Tue, 13 Aug 2019 21:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFBAE6B0296; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CDDF76B029A; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 931EB6B029B; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 6729C6B0296 for ; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 046D32C34 for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-FDA: 75818631084.30.cow85_2034196105a2f X-HE-Tag: cow85_2034196105a2f X-Filterd-Recvd-Size: 7942 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:41 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275965" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:39 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 02/14] x86/cet/ibt: User-mode indirect branch tracking support Date: Tue, 13 Aug 2019 13:53:47 -0700 Message-Id: <20190813205359.12196-3-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add user-mode indirect branch tracking enabling/disabling and supporting routines. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 6 ++++ arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/kernel/cet.c | 36 +++++++++++++++++++ arch/x86/kernel/cpu/common.c | 17 +++++++++ .../arch/x86/include/asm/disabled-features.h | 8 ++++- 5 files changed, 73 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 2df357dffd24..2561efe081ad 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -16,6 +16,8 @@ struct cet_status { unsigned long shstk_size; unsigned int locked:1; unsigned int shstk_enabled:1; + unsigned int ibt_enabled:1; + unsigned int ibt_bitmap_used:1; }; #ifdef CONFIG_X86_INTEL_CET @@ -27,6 +29,8 @@ void cet_disable_shstk(void); void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(bool ia32, struct sc_ext *sc); int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); +int cet_setup_ibt(void); +void cet_disable_ibt(void); #else static inline int prctl_cet(int option, unsigned long arg2) { return -EINVAL; } static inline int cet_setup_shstk(void) { return -EINVAL; } @@ -37,6 +41,8 @@ static inline void cet_disable_free_shstk(struct task_struct *p) {} static inline int cet_restore_signal(bool ia32, struct sc_ext *sc) { return -EINVAL; } static inline int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc) { return -EINVAL; } +static inline int cet_setup_ibt(void) { return -EINVAL; } +static inline void cet_disable_ibt(void) {} #endif #define cpu_x86_cet_enabled() \ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 06323ebed643..fc7d3d5a1bf4 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -68,6 +68,12 @@ #define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) #endif +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1<<(X86_FEATURE_IBT & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -89,7 +95,7 @@ #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) #endif /* _ASM_X86_DISABLED_FEATURES_H */ diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index e4e20d6ab07b..505a69f476e1 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -13,6 +13,8 @@ #include #include #include +#include +#include #include #include #include @@ -325,3 +327,37 @@ int cet_setup_signal(bool ia32, unsigned long rstor_addr, struct sc_ext *sc_ext) modify_fpu_regs_end(); return 0; } + +int cet_setup_ibt(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_IBT)) + return -EOPNOTSUPP; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, r); + r |= (MSR_IA32_CET_ENDBR_EN | MSR_IA32_CET_NO_TRACK_EN); + wrmsrl(MSR_IA32_U_CET, r); + modify_fpu_regs_end(); + + current->thread.cet.ibt_enabled = 1; + return 0; +} + +void cet_disable_ibt(void) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_IBT)) + return; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, r); + r &= ~(MSR_IA32_CET_ENDBR_EN | MSR_IA32_CET_LEG_IW_EN | + MSR_IA32_CET_NO_TRACK_EN | MSR_IA32_CET_BITMAP_MASK); + wrmsrl(MSR_IA32_U_CET, r); + modify_fpu_regs_end(); + + current->thread.cet.ibt_enabled = 0; +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index d3addbd3f4d4..092979715d16 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -512,6 +512,23 @@ static __init int setup_disable_shstk(char *s) __setup("no_cet_shstk", setup_disable_shstk); #endif +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER +static __init int setup_disable_ibt(char *s) +{ + /* require an exact match without trailing characters */ + if (s[0] != '\0') + return 0; + + if (!boot_cpu_has(X86_FEATURE_IBT)) + return 1; + + setup_clear_cpu_cap(X86_FEATURE_IBT); + pr_info("x86: 'no_cet_ibt' specified, disabling Branch Tracking\n"); + return 1; +} +__setup("no_cet_ibt", setup_disable_ibt); +#endif + /* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization diff --git a/tools/arch/x86/include/asm/disabled-features.h b/tools/arch/x86/include/asm/disabled-features.h index 06323ebed643..fc7d3d5a1bf4 100644 --- a/tools/arch/x86/include/asm/disabled-features.h +++ b/tools/arch/x86/include/asm/disabled-features.h @@ -68,6 +68,12 @@ #define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) #endif +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1<<(X86_FEATURE_IBT & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -89,7 +95,7 @@ #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK) #define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) #endif /* _ASM_X86_DISABLED_FEATURES_H */ From patchwork Tue Aug 13 20:53:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092899 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C384912 for ; Tue, 13 Aug 2019 21:04:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E065286BF for ; Tue, 13 Aug 2019 21:04:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 31E4528718; Tue, 13 Aug 2019 21:04:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB34B2868D for ; Tue, 13 Aug 2019 21:04:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B8406B0299; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 62FCB6B029B; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38D566B029E; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id EA4036B0299 for ; Tue, 13 Aug 2019 17:03:42 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 8B1B1180AD7C1 for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-FDA: 75818631084.02.screw42_205168e7e4020 X-HE-Tag: screw42_205168e7e4020 X-Filterd-Recvd-Size: 3496 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:41 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275968" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:39 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 03/14] x86/cet/ibt: Handle signals for end branch Date: Tue, 13 Aug 2019 13:53:48 -0700 Message-Id: <20190813205359.12196-4-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Restore end branch tracking setting from thread header. Signed-off-by: Yu-cheng Yu --- arch/x86/kernel/cet.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/kernel/cet.c b/arch/x86/kernel/cet.c index 505a69f476e1..db542bd423cc 100644 --- a/arch/x86/kernel/cet.c +++ b/arch/x86/kernel/cet.c @@ -281,6 +281,15 @@ int cet_restore_signal(bool ia32, struct sc_ext *sc_ext) msr_ia32_u_cet |= MSR_IA32_CET_SHSTK_EN; } + if (current->thread.cet.ibt_enabled) { + if (current->thread.cet.ibt_bitmap_used) + msr_ia32_u_cet |= (IBT_BITMAP_ADDR | + MSR_IA32_CET_LEG_IW_EN); + + msr_ia32_u_cet |= (MSR_IA32_CET_ENDBR_EN | + MSR_IA32_CET_NO_TRACK_EN); + } + wrmsrl(MSR_IA32_PL3_SSP, new_ssp); wrmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); return 0; @@ -321,6 +330,15 @@ int cet_setup_signal(bool ia32, unsigned long rstor_addr, struct sc_ext *sc_ext) sc_ext->ssp = new_ssp; } + if (current->thread.cet.ibt_enabled) { + if (current->thread.cet.ibt_bitmap_used) + msr_ia32_u_cet |= (IBT_BITMAP_ADDR | + MSR_IA32_CET_LEG_IW_EN); + + msr_ia32_u_cet |= (MSR_IA32_CET_ENDBR_EN | + MSR_IA32_CET_NO_TRACK_EN); + } + modify_fpu_regs_begin(); wrmsrl(MSR_IA32_PL3_SSP, ssp); wrmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); From patchwork Tue Aug 13 20:53:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 588481399 for ; Tue, 13 Aug 2019 21:04:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A7682868D for ; Tue, 13 Aug 2019 21:04:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 33E512873A; Tue, 13 Aug 2019 21:04:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4CA1286E0 for ; Tue, 13 Aug 2019 21:04:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A23066B029B; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9AB296B02A0; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 477236B029D; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id 0CA986B029B for ; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id A2B56180AD7C3 for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-FDA: 75818631084.21.ball38_2051f0cbb3024 X-HE-Tag: ball38_2051f0cbb3024 X-Filterd-Recvd-Size: 4176 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:41 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275971" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:39 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 04/14] mm/mmap: Add IBT bitmap size to address space limit check Date: Tue, 13 Aug 2019 13:53:49 -0700 Message-Id: <20190813205359.12196-5-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The indirect branch tracking legacy bitmap takes a large address space. This causes may_expand_vm() failure on the address limit check. For a IBT-enabled task, add the bitmap size to the address limit. Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/mmu_context.h | 10 ++++++++++ mm/mmap.c | 19 ++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index a9a768529540..e18e89ef332d 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -360,6 +360,16 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER +static inline unsigned long arch_as_limit(void) +{ + if (current->thread.cet.ibt_enabled) + return IBT_BITMAP_SIZE; + else + return 0; +} +#endif + typedef struct { struct mm_struct *mm; } temp_mm_state_t; diff --git a/mm/mmap.c b/mm/mmap.c index 1acded00f003..1bfd9aec9aed 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3283,13 +3283,30 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, return NULL; } +#ifndef CONFIG_ARCH_HAS_AS_LIMIT +static inline unsigned long arch_as_limit(void) +{ + return 0; +} +#endif + /* * Return true if the calling process may expand its vm space by the passed * number of pages */ bool may_expand_vm(struct mm_struct *mm, vm_flags_t flags, unsigned long npages) { - if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> PAGE_SHIFT) + unsigned long as_limit = rlimit(RLIMIT_AS); + unsigned long as_limit_plus = as_limit + arch_as_limit(); + + /* as_limit_plus overflowed */ + if (as_limit_plus < as_limit) + as_limit_plus = RLIM_INFINITY; + + if (as_limit_plus > as_limit) + as_limit = as_limit_plus; + + if (mm->total_vm + npages > as_limit >> PAGE_SHIFT) return false; if (is_data_mapping(flags) && From patchwork Tue Aug 13 20:53:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3843A1399 for ; Tue, 13 Aug 2019 21:04:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 283192871C for ; Tue, 13 Aug 2019 21:04:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1BFD0286E0; Tue, 13 Aug 2019 21:04:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA6BA286D1 for ; Tue, 13 Aug 2019 21:04:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9D616B029E; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D09126B029F; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8784C6B02A1; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 42EEB6B029F for ; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id E0C5A8248AA1 for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-FDA: 75818631084.08.owl02_205d5799c5533 X-HE-Tag: owl02_205d5799c5533 X-Filterd-Recvd-Size: 3338 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275975" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 05/14] x86/cet/ibt: ELF header parsing for IBT Date: Tue, 13 Aug 2019 13:53:50 -0700 Message-Id: <20190813205359.12196-6-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Look in .note.gnu.property of an ELF file and check if Indirect Branch Tracking needs to be enabled for the task. Signed-off-by: Yu-cheng Yu --- arch/x86/Kconfig | 1 + arch/x86/kernel/process_64.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 666dc66a382e..7edb544cef00 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1971,6 +1971,7 @@ config X86_INTEL_BRANCH_TRACKING_USER select X86_INTEL_CET select ARCH_HAS_AS_LIMIT select ARCH_USE_GNU_PROPERTY + select ARCH_BINFMT_ELF_STATE ---help--- Indirect Branch Tracking provides hardware protection against return-/jmp- oriented programming attacks. diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 7ec60b14e96d..a051cfc5242f 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -856,6 +856,11 @@ int arch_setup_property(struct arch_elf_state *state) return r; } + if (cpu_feature_enabled(X86_FEATURE_IBT)) { + if (state->gnu_property & GNU_PROPERTY_X86_FEATURE_1_IBT) + r = cet_setup_ibt(); + } + return r; } #endif From patchwork Tue Aug 13 20:53:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18BE2912 for ; Tue, 13 Aug 2019 21:04:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0AA62286BF for ; Tue, 13 Aug 2019 21:04:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F18C52873A; Tue, 13 Aug 2019 21:04:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80F3D286E0 for ; Tue, 13 Aug 2019 21:04:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BAB46B02A4; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 257F26B02A1; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 036676B02A4; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id AA16D6B029D for ; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 517642C34 for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-FDA: 75818631126.26.year03_206dddc91bd24 X-HE-Tag: year03_206dddc91bd24 X-Filterd-Recvd-Size: 3829 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275979" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 06/14] x86/cet/ibt: Add arch_prctl functions for IBT Date: Tue, 13 Aug 2019 13:53:51 -0700 Message-Id: <20190813205359.12196-7-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "H.J. Lu" Update ARCH_X86_CET_STATUS and ARCH_X86_CET_DISABLE to include Indirect Branch Tracking features. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/include/uapi/asm/prctl.h | 2 ++ arch/x86/kernel/cet_prctl.c | 4 ++++ 2 files changed, 6 insertions(+) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index d962f0ec9ccf..02243127dcf6 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -18,5 +18,7 @@ #define ARCH_X86_CET_DISABLE 0x3002 #define ARCH_X86_CET_LOCK 0x3003 #define ARCH_X86_CET_ALLOC_SHSTK 0x3004 +#define ARCH_X86_CET_GET_LEGACY_BITMAP 0x3005 /* deprecated */ +#define ARCH_X86_CET_SET_LEGACY_BITMAP 0x3006 /* deprecated */ #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c index 9c9d4262b07e..09d8c4ea935c 100644 --- a/arch/x86/kernel/cet_prctl.c +++ b/arch/x86/kernel/cet_prctl.c @@ -20,6 +20,8 @@ static int handle_get_status(unsigned long arg2) if (current->thread.cet.shstk_enabled) features |= GNU_PROPERTY_X86_FEATURE_1_SHSTK; + if (current->thread.cet.ibt_enabled) + features |= GNU_PROPERTY_X86_FEATURE_1_IBT; shstk_base = current->thread.cet.shstk_base; shstk_size = current->thread.cet.shstk_size; @@ -69,6 +71,8 @@ int prctl_cet(int option, unsigned long arg2) return -EPERM; if (arg2 & GNU_PROPERTY_X86_FEATURE_1_SHSTK) cet_disable_free_shstk(current); + if (arg2 & GNU_PROPERTY_X86_FEATURE_1_IBT) + cet_disable_ibt(); return 0; From patchwork Tue Aug 13 20:53:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 140691399 for ; Tue, 13 Aug 2019 21:04:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 04A7A2868D for ; Tue, 13 Aug 2019 21:04:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EC7272871C; Tue, 13 Aug 2019 21:04:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D9DE2868D for ; Tue, 13 Aug 2019 21:04:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8985B6B02A6; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7ACBB6B029D; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BA236B029D; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id DB07F6B02A0 for ; Tue, 13 Aug 2019 17:03:43 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 87CBC55F98 for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-FDA: 75818631126.18.crate76_2070959672427 X-HE-Tag: crate76_2070959672427 X-Filterd-Recvd-Size: 4406 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:42 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275982" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 07/14] x86/cet/ibt: Add ENDBR to op-code-map Date: Tue, 13 Aug 2019 13:53:52 -0700 Message-Id: <20190813205359.12196-8-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add control transfer terminating instructions: ENDBR64/ENDBR32: Mark a valid 64/32-bit control transfer endpoint. Signed-off-by: Yu-cheng Yu --- arch/x86/lib/x86-opcode-map.txt | 13 +++++++++++-- tools/objtool/arch/x86/lib/x86-opcode-map.txt | 13 +++++++++++-- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/arch/x86/lib/x86-opcode-map.txt b/arch/x86/lib/x86-opcode-map.txt index c5e825d44766..fbc53481bc59 100644 --- a/arch/x86/lib/x86-opcode-map.txt +++ b/arch/x86/lib/x86-opcode-map.txt @@ -620,7 +620,16 @@ ea: SAVEPREVSSP (f3) # Skip 0xeb-0xff EndTable -Table: 3-byte opcode 2 (0x0f 0x38) +Table: 3-byte opcode 2 (0x0f 0x1e) +Referrer: +AVXcode: +# Skip 0x00-0xf9 +fa: ENDBR64 (f3) +fb: ENDBR32 (f3) +#skip 0xfc-0xff +EndTable + +Table: 3-byte opcode 3 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -804,7 +813,7 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 3 (0x0f 0x3a) +Table: 3-byte opcode 4 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff diff --git a/tools/objtool/arch/x86/lib/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt index c5e825d44766..fbc53481bc59 100644 --- a/tools/objtool/arch/x86/lib/x86-opcode-map.txt +++ b/tools/objtool/arch/x86/lib/x86-opcode-map.txt @@ -620,7 +620,16 @@ ea: SAVEPREVSSP (f3) # Skip 0xeb-0xff EndTable -Table: 3-byte opcode 2 (0x0f 0x38) +Table: 3-byte opcode 2 (0x0f 0x1e) +Referrer: +AVXcode: +# Skip 0x00-0xf9 +fa: ENDBR64 (f3) +fb: ENDBR32 (f3) +#skip 0xfc-0xff +EndTable + +Table: 3-byte opcode 3 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f @@ -804,7 +813,7 @@ f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 3 (0x0f 0x3a) +Table: 3-byte opcode 4 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff From patchwork Tue Aug 13 20:53:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FD7A1399 for ; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 906202868D for ; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7F91728748; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 878442873A for ; Tue, 13 Aug 2019 21:04:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B544E6B029D; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 896546B02A1; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 737756B02A7; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 309176B02A0 for ; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id CAB27181AC9B4 for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-FDA: 75818631126.21.basin66_207b2b2d52137 X-HE-Tag: basin66_207b2b2d52137 X-Filterd-Recvd-Size: 4484 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275985" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:40 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 08/14] x86/vdso: Insert endbr32/endbr64 to vDSO Date: Tue, 13 Aug 2019 13:53:53 -0700 Message-Id: <20190813205359.12196-9-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "H.J. Lu" When Intel indirect branch tracking is enabled, functions in vDSO which may be called indirectly must have endbr32 or endbr64 as the first instruction. Compiler must support -fcf-protection=branch so that it can be used to compile vDSO. Acked-by: Andy Lutomirski Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/entry/vdso/Makefile | 12 +++++++++++- arch/x86/entry/vdso/vdso-layout.lds.S | 1 + 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index 8df549138193..1e6a95881e73 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -114,13 +114,17 @@ vobjx32s := $(foreach F,$(vobjx32s-y),$(obj)/$F) # Convert 64bit object file to x32 for x32 vDSO. quiet_cmd_x32 = X32 $@ - cmd_x32 = $(OBJCOPY) -O elf32-x86-64 $< $@ + cmd_x32 = $(OBJCOPY) -R .note.gnu.property -O elf32-x86-64 $< $@ $(obj)/%-x32.o: $(obj)/%.o FORCE $(call if_changed,x32) targets += vdsox32.lds $(vobjx32s-y) +ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + $(obj)/vclock_gettime.o $(obj)/vgetcpu.o $(obj)/vdso32/vclock_gettime.o: KBUILD_CFLAGS += -fcf-protection=branch +endif + $(obj)/%.so: OBJCOPYFLAGS := -S $(obj)/%.so: $(obj)/%.so.dbg FORCE $(call if_changed,objcopy) @@ -178,6 +182,12 @@ quiet_cmd_vdso = VDSO $@ VDSO_LDFLAGS = -shared --hash-style=both --build-id \ $(call ld-option, --eh-frame-hdr) -Bsymbolic +ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + VDSO_LDFLAGS += $(call ldoption, -z$(comma)ibt) +endif +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + VDSO_LDFLAGS += $(call ldoption, -z$(comma)shstk) +endif GCOV_PROFILE := n quiet_cmd_vdso_and_check = VDSO $@ diff --git a/arch/x86/entry/vdso/vdso-layout.lds.S b/arch/x86/entry/vdso/vdso-layout.lds.S index 93c6dc7812d0..3fea2ce318bc 100644 --- a/arch/x86/entry/vdso/vdso-layout.lds.S +++ b/arch/x86/entry/vdso/vdso-layout.lds.S @@ -52,6 +52,7 @@ SECTIONS *(.gnu.linkonce.b.*) } :text + .note.gnu.property : { *(.note.gnu.property) } :text :note .note : { *(.note.*) } :text :note .eh_frame_hdr : { *(.eh_frame_hdr) } :text :eh_frame_hdr From patchwork Tue Aug 13 20:53:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED0D9912 for ; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDE972868D for ; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CF376286E0; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 759C828746 for ; Tue, 13 Aug 2019 21:04:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22E016B02A0; Tue, 13 Aug 2019 17:03:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1B8026B02A8; Tue, 13 Aug 2019 17:03:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFE0E6B02AA; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id 9D1286B02A0 for ; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 452358248AA2 for ; Tue, 13 Aug 2019 21:03:44 +0000 (UTC) X-FDA: 75818631168.08.pear36_209212ba6022a X-HE-Tag: pear36_209212ba6022a X-Filterd-Recvd-Size: 2966 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275988" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 09/14] x86/vdso/32: Add ENDBR32 to __kernel_vsyscall entry point Date: Tue, 13 Aug 2019 13:53:54 -0700 Message-Id: <20190813205359.12196-10-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "H.J. Lu" Add ENDBR32 to __kernel_vsyscall entry point. Acked-by: Andy Lutomirski Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/entry/vdso/vdso32/system_call.S | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/entry/vdso/vdso32/system_call.S b/arch/x86/entry/vdso/vdso32/system_call.S index 263d7433dea8..2fc8141fff4e 100644 --- a/arch/x86/entry/vdso/vdso32/system_call.S +++ b/arch/x86/entry/vdso/vdso32/system_call.S @@ -14,6 +14,9 @@ ALIGN __kernel_vsyscall: CFI_STARTPROC +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + endbr32 +#endif /* * Reshuffle regs so that all of any of the entry instructions * will preserve enough state. From patchwork Tue Aug 13 20:53:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0793F1399 for ; Tue, 13 Aug 2019 21:04:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC56D286E0 for ; Tue, 13 Aug 2019 21:04:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DFCD228741; Tue, 13 Aug 2019 21:04:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D0D92867F for ; Tue, 13 Aug 2019 21:04:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F7F36B02A1; Tue, 13 Aug 2019 17:03:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 42DDE6B02AB; Tue, 13 Aug 2019 17:03:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0803B6B02A1; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id C36736B02A8 for ; Tue, 13 Aug 2019 17:03:44 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 69742181AC9B4 for ; Tue, 13 Aug 2019 21:03:44 +0000 (UTC) X-FDA: 75818631168.29.pen56_2093cacaa582a X-HE-Tag: pen56_2093cacaa582a X-Filterd-Recvd-Size: 3210 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275993" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 10/14] x86/vsyscall/64: Add ENDBR64 to vsyscall entry points Date: Tue, 13 Aug 2019 13:53:55 -0700 Message-Id: <20190813205359.12196-11-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "H.J. Lu" Add ENDBR64 to vsyscall entry points. Acked-by: Andy Lutomirski Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/entry/vsyscall/vsyscall_emu_64.S | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/entry/vsyscall/vsyscall_emu_64.S b/arch/x86/entry/vsyscall/vsyscall_emu_64.S index 2e203f3a25a7..040696333457 100644 --- a/arch/x86/entry/vsyscall/vsyscall_emu_64.S +++ b/arch/x86/entry/vsyscall/vsyscall_emu_64.S @@ -17,16 +17,25 @@ __PAGE_ALIGNED_DATA .type __vsyscall_page, @object __vsyscall_page: +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + endbr64 +#endif mov $__NR_gettimeofday, %rax syscall ret .balign 1024, 0xcc +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + endbr64 +#endif mov $__NR_time, %rax syscall ret .balign 1024, 0xcc +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + endbr64 +#endif mov $__NR_getcpu, %rax syscall ret From patchwork Tue Aug 13 20:53:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092917 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1473A1399 for ; Tue, 13 Aug 2019 21:04:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 044AC28718 for ; Tue, 13 Aug 2019 21:04:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ECA1B2871C; Tue, 13 Aug 2019 21:04:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 884E52873A for ; Tue, 13 Aug 2019 21:04:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C1256B000A; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6770A6B02AA; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B48B6B02AB; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0D93A6B000A for ; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id A3FD48248AA2 for ; Tue, 13 Aug 2019 21:03:45 +0000 (UTC) X-FDA: 75818631210.07.ship94_209cfd6a0c93c X-HE-Tag: ship94_209cfd6a0c93c X-Filterd-Recvd-Size: 4223 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:43 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194275998" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:41 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 11/14] x86/vsyscall/64: Fixup shadow stack and branch tracking for vsyscall Date: Tue, 13 Aug 2019 13:53:56 -0700 Message-Id: <20190813205359.12196-12-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When emulating a RET, also unwind the task's shadow stack and cancel the current branch tracking status. Signed-off-by: Yu-cheng Yu --- arch/x86/entry/vsyscall/vsyscall_64.c | 29 ++++++++++++++++++++++++ arch/x86/entry/vsyscall/vsyscall_trace.h | 1 + 2 files changed, 30 insertions(+) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index e7c596dea947..27ff81f75c82 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -38,6 +38,9 @@ #include #include #include +#include +#include +#include #define CREATE_TRACE_POINTS #include "vsyscall_trace.h" @@ -286,6 +289,32 @@ bool emulate_vsyscall(unsigned long error_code, /* Emulate a ret instruction. */ regs->ip = caller; regs->sp += 8; + + /* Unwind shadow stack. */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + if (current->thread.cet.shstk_enabled) { + u64 r; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_PL3_SSP, r); + wrmsrl(MSR_IA32_PL3_SSP, r + 8); + modify_fpu_regs_end(); + } +#endif + + /* Fixup branch tracking */ +#ifdef CONFIG_X86_INTEL_BRANCH_TRACKING_USER + if (current->thread.cet.ibt_enabled) { + u64 r; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_U_CET, r & ~MSR_IA32_CET_WAIT_ENDBR); + modify_fpu_regs_end(); + } +#endif + return true; sigsegv: diff --git a/arch/x86/entry/vsyscall/vsyscall_trace.h b/arch/x86/entry/vsyscall/vsyscall_trace.h index 3c3f9765a85c..7aa2101ada44 100644 --- a/arch/x86/entry/vsyscall/vsyscall_trace.h +++ b/arch/x86/entry/vsyscall/vsyscall_trace.h @@ -25,6 +25,7 @@ TRACE_EVENT(emulate_vsyscall, #endif #undef TRACE_INCLUDE_PATH +#undef TRACE_INCLUDE_FILE #define TRACE_INCLUDE_PATH ../../arch/x86/entry/vsyscall/ #define TRACE_INCLUDE_FILE vsyscall_trace #include From patchwork Tue Aug 13 20:53:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7192B1399 for ; Tue, 13 Aug 2019 21:04:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 635C72867F for ; Tue, 13 Aug 2019 21:04:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 56A8D2871C; Tue, 13 Aug 2019 21:04:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA01D2867F for ; Tue, 13 Aug 2019 21:04:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEF976B000D; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D02416B02AD; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B564A6B02AF; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0238.hostedemail.com [216.40.44.238]) by kanga.kvack.org (Postfix) with ESMTP id 637C86B000D for ; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 0247B180AD7C1 for ; Tue, 13 Aug 2019 21:03:46 +0000 (UTC) X-FDA: 75818631252.11.rose43_20c59a923cb30 X-HE-Tag: rose43_20c59a923cb30 X-Filterd-Recvd-Size: 7047 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:45 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194276001" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 12/14] x86/cet: Add PTRACE interface for CET Date: Tue, 13 Aug 2019 13:53:57 -0700 Message-Id: <20190813205359.12196-13-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add REGSET_CET64/REGSET_CET32 to get/set CET MSRs: IA32_U_CET (user-mode CET settings) and IA32_PL3_SSP (user-mode shadow stack) Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/fpu/regset.h | 7 +++--- arch/x86/kernel/fpu/regset.c | 41 +++++++++++++++++++++++++++++++ arch/x86/kernel/ptrace.c | 16 ++++++++++++ include/uapi/linux/elf.h | 1 + 4 files changed, 62 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/fpu/regset.h b/arch/x86/include/asm/fpu/regset.h index d5bdffb9d27f..edad0d889084 100644 --- a/arch/x86/include/asm/fpu/regset.h +++ b/arch/x86/include/asm/fpu/regset.h @@ -7,11 +7,12 @@ #include -extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active; +extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active, + cetregs_active; extern user_regset_get_fn fpregs_get, xfpregs_get, fpregs_soft_get, - xstateregs_get; + xstateregs_get, cetregs_get; extern user_regset_set_fn fpregs_set, xfpregs_set, fpregs_soft_set, - xstateregs_set; + xstateregs_set, cetregs_set; /* * xstateregs_active == regset_fpregs_active. Please refer to the comment diff --git a/arch/x86/kernel/fpu/regset.c b/arch/x86/kernel/fpu/regset.c index d652b939ccfb..2937ec9d9215 100644 --- a/arch/x86/kernel/fpu/regset.c +++ b/arch/x86/kernel/fpu/regset.c @@ -156,6 +156,47 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, return ret; } +int cetregs_active(struct task_struct *target, const struct user_regset *regset) +{ +#ifdef CONFIG_X86_INTEL_CET + if (target->thread.cet.shstk_enabled || target->thread.cet.ibt_enabled) + return regset->n; +#endif + return 0; +} + +int cetregs_get(struct task_struct *target, const struct user_regset *regset, + unsigned int pos, unsigned int count, + void *kbuf, void __user *ubuf) +{ + struct fpu *fpu = &target->thread.fpu; + struct cet_user_state *cetregs; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return -ENODEV; + + cetregs = get_xsave_addr(&fpu->state.xsave, XFEATURE_CET_USER); + + fpu__prepare_read(fpu); + return user_regset_copyout(&pos, &count, &kbuf, &ubuf, cetregs, 0, -1); +} + +int cetregs_set(struct task_struct *target, const struct user_regset *regset, + unsigned int pos, unsigned int count, + const void *kbuf, const void __user *ubuf) +{ + struct fpu *fpu = &target->thread.fpu; + struct cet_user_state *cetregs; + + if (!boot_cpu_has(X86_FEATURE_SHSTK)) + return -ENODEV; + + cetregs = get_xsave_addr(&fpu->state.xsave, XFEATURE_CET_USER); + + fpu__prepare_write(fpu); + return user_regset_copyin(&pos, &count, &kbuf, &ubuf, cetregs, 0, -1); +} + #if defined CONFIG_X86_32 || defined CONFIG_IA32_EMULATION /* diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c index 3c5bbe8e4120..4bae0faa5331 100644 --- a/arch/x86/kernel/ptrace.c +++ b/arch/x86/kernel/ptrace.c @@ -52,7 +52,9 @@ enum x86_regset { REGSET_IOPERM64 = REGSET_XFP, REGSET_XSTATE, REGSET_TLS, + REGSET_CET64 = REGSET_TLS, REGSET_IOPERM32, + REGSET_CET32, }; struct pt_regs_offset { @@ -1239,6 +1241,13 @@ static struct user_regset x86_64_regsets[] __ro_after_init = { .size = sizeof(long), .align = sizeof(long), .active = ioperm_active, .get = ioperm_get }, + [REGSET_CET64] = { + .core_note_type = NT_X86_CET, + .n = sizeof(struct cet_user_state) / sizeof(u64), + .size = sizeof(u64), .align = sizeof(u64), + .active = cetregs_active, .get = cetregs_get, + .set = cetregs_set + }, }; static const struct user_regset_view user_x86_64_view = { @@ -1294,6 +1303,13 @@ static struct user_regset x86_32_regsets[] __ro_after_init = { .size = sizeof(u32), .align = sizeof(u32), .active = ioperm_active, .get = ioperm_get }, + [REGSET_CET32] = { + .core_note_type = NT_X86_CET, + .n = sizeof(struct cet_user_state) / sizeof(u64), + .size = sizeof(u64), .align = sizeof(u64), + .active = cetregs_active, .get = cetregs_get, + .set = cetregs_set + }, }; static const struct user_regset_view user_x86_32_view = { diff --git a/include/uapi/linux/elf.h b/include/uapi/linux/elf.h index 530ce08467c2..349c435a2ce1 100644 --- a/include/uapi/linux/elf.h +++ b/include/uapi/linux/elf.h @@ -401,6 +401,7 @@ typedef struct elf64_shdr { #define NT_386_TLS 0x200 /* i386 TLS slots (struct user_desc) */ #define NT_386_IOPERM 0x201 /* x86 io permission bitmap (1=deny) */ #define NT_X86_XSTATE 0x202 /* x86 extended state using xsave */ +#define NT_X86_CET 0x203 /* x86 cet state */ #define NT_S390_HIGH_GPRS 0x300 /* s390 upper register halves */ #define NT_S390_TIMER 0x301 /* s390 timer register */ #define NT_S390_TODCMP 0x302 /* s390 TOD clock comparator register */ From patchwork Tue Aug 13 20:53:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21630912 for ; Tue, 13 Aug 2019 21:04:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 12CDA2867F for ; Tue, 13 Aug 2019 21:04:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0600728748; Tue, 13 Aug 2019 21:04:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99F6128741 for ; Tue, 13 Aug 2019 21:04:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 982CF6B02AB; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7B8AA6B02AD; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58AFC6B02AD; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id 280796B000D for ; Tue, 13 Aug 2019 17:03:46 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id C0D4C4FFA for ; Tue, 13 Aug 2019 21:03:45 +0000 (UTC) X-FDA: 75818631210.13.scarf46_20c5410cb7830 X-HE-Tag: scarf46_20c5410cb7830 X-Filterd-Recvd-Size: 4033 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:45 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194276005" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 13/14] x86: Discard .note.gnu.property sections Date: Tue, 13 Aug 2019 13:53:58 -0700 Message-Id: <20190813205359.12196-14-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: "H.J. Lu" With the command-line option, -mx86-used-note=yes, the x86 assembler in binutils 2.32 and above generates a program property note in a note section, .note.gnu.property, to encode used x86 ISAs and features. To exclude .note.gnu.property sections from NOTE segment in x86 kernel linker script: PHDRS { text PT_LOAD FLAGS(5); data PT_LOAD FLAGS(6); percpu PT_LOAD FLAGS(6); init PT_LOAD FLAGS(7); note PT_NOTE FLAGS(0); } SECTIONS { ... .notes : AT(ADDR(.notes) - 0xffffffff80000000) { __start_notes = .; KEEP(*(.not e.*)) __stop_notes = .; } :text :note ... } this patch discards .note.gnu.property sections in kernel linker script by adding /DISCARD/ : { *(.note.gnu.property) } before .notes sections. Since .exit.text and .exit.data sections are discarded at runtime, it undefines EXIT_TEXT and EXIT_DATA to exclude .exit.text and .exit.data sections from default discarded sections. Signed-off-by: H.J. Lu Signed-off-by: Yu-cheng Yu --- arch/x86/kernel/vmlinux.lds.S | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index e2feacf921a0..5ef137493a85 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -146,6 +146,10 @@ SECTIONS _etext = .; } :text = 0x9090 + /* .note.gnu.property sections should be discarded */ + /DISCARD/ : { + *(.note.gnu.property) + } NOTES :text :note EXCEPTION_TABLE(16) :text = 0x9090 @@ -415,6 +419,12 @@ SECTIONS STABS_DEBUG DWARF_DEBUG + /* Sections to be discarded. EXIT_TEXT and EXIT_DATA discard at + * runtime, not link time. */ +#undef EXIT_TEXT +#define EXIT_TEXT +#undef EXIT_DATA +#define EXIT_DATA DISCARDS /DISCARD/ : { *(.eh_frame) From patchwork Tue Aug 13 20:53:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu-cheng Yu X-Patchwork-Id: 11092923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 476A91399 for ; Tue, 13 Aug 2019 21:04:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 36A1528718 for ; Tue, 13 Aug 2019 21:04:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A58328671; Tue, 13 Aug 2019 21:04:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A4392871C for ; Tue, 13 Aug 2019 21:04:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA69E6B02AD; Tue, 13 Aug 2019 17:03:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2AC96B02AF; Tue, 13 Aug 2019 17:03:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B30BA6B02B1; Tue, 13 Aug 2019 17:03:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 7EB8B6B02AD for ; Tue, 13 Aug 2019 17:03:47 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 24A64180AD7C1 for ; Tue, 13 Aug 2019 21:03:47 +0000 (UTC) X-FDA: 75818631294.08.hall48_20f452c5ad442 X-HE-Tag: hall48_20f452c5ad442 X-Filterd-Recvd-Size: 13402 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 13 Aug 2019 21:03:46 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 14:03:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,382,1559545200"; d="scan'208";a="194276008" Received: from yyu32-desk1.sc.intel.com ([10.144.153.205]) by fmsmga001.fm.intel.com with ESMTP; 13 Aug 2019 14:03:42 -0700 From: Yu-cheng Yu To: x86@kernel.org, "H. Peter Anvin" , Thomas Gleixner , Ingo Molnar , linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-api@vger.kernel.org, Arnd Bergmann , Andy Lutomirski , Balbir Singh , Borislav Petkov , Cyrill Gorcunov , Dave Hansen , Eugene Syromiatnikov , Florian Weimer , "H.J. Lu" , Jann Horn , Jonathan Corbet , Kees Cook , Mike Kravetz , Nadav Amit , Oleg Nesterov , Pavel Machek , Peter Zijlstra , Randy Dunlap , "Ravi V. Shankar" , Vedvyas Shanbhogue , Dave Martin Cc: Yu-cheng Yu Subject: [PATCH v8 14/14] Introduce arch_prctl(ARCH_X86_CET_MARK_LEGACY_CODE) Date: Tue, 13 Aug 2019 13:53:59 -0700 Message-Id: <20190813205359.12196-15-yu-cheng.yu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190813205359.12196-1-yu-cheng.yu@intel.com> References: <20190813205359.12196-1-yu-cheng.yu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When CET Indirect Branch Tracking (IBT) is enabled, the processor expects every branch target is an ENDBR instruction, or the target's address is marked as legacy in the legacy code bitmap. The bitmap covers the whole user-mode address space (TASK_SIZE_MAX for 64-bit, TASK_SIZE for IA32), and each bit represents one page of linear address range. The bitmap is located at the topmost address: (TASK_SIZE - IBT_BITMAP_SIZE). It is allocated only when the first time ARCH_X86_MARK_LEGACY_CODE is called from an application. The IBT bitmap is visiable from user-mode, but not writable. Introduce: arch_prctl(ARCH_X86_CET_MARK_LEGACY_CODE, unsigned long *buf) Mark an address range as IBT legacy code. *buf: starting linear address *(buf + 1): size of the legacy code *(buf + 2): set (1); clear (0) Signed-off-by: Yu-cheng Yu --- arch/x86/include/asm/cet.h | 3 + arch/x86/include/asm/processor.h | 13 +- arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet_bitmap.c | 210 ++++++++++++++++++++++++++++++ arch/x86/kernel/cet_prctl.c | 15 +++ mm/memory.c | 8 ++ 7 files changed, 250 insertions(+), 2 deletions(-) create mode 100644 arch/x86/kernel/cet_bitmap.c diff --git a/arch/x86/include/asm/cet.h b/arch/x86/include/asm/cet.h index 2561efe081ad..d5f693d082b0 100644 --- a/arch/x86/include/asm/cet.h +++ b/arch/x86/include/asm/cet.h @@ -4,6 +4,7 @@ #ifndef __ASSEMBLY__ #include +#include struct task_struct; struct sc_ext; @@ -30,6 +31,7 @@ void cet_disable_free_shstk(struct task_struct *p); int cet_restore_signal(bool ia32, struct sc_ext *sc); int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc); int cet_setup_ibt(void); +int cet_mark_legacy_code(unsigned long addr, unsigned long size, unsigned long set); void cet_disable_ibt(void); #else static inline int prctl_cet(int option, unsigned long arg2) { return -EINVAL; } @@ -42,6 +44,7 @@ static inline int cet_restore_signal(bool ia32, struct sc_ext *sc) { return -EIN static inline int cet_setup_signal(bool ia32, unsigned long rstor, struct sc_ext *sc) { return -EINVAL; } static inline int cet_setup_ibt(void) { return -EINVAL; } +static inline int cet_mark_legacy_code(unsigned long addr, unsigned long size, unsigned long set) { return -EINVAL; } static inline void cet_disable_ibt(void) {} #endif diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 0f9bc7fd1351..af3bdd545a55 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -888,7 +888,18 @@ static inline void spin_lock_prefetch(const void *x) #define TASK_SIZE_OF(child) ((test_tsk_thread_flag(child, TIF_ADDR32)) ? \ IA32_PAGE_OFFSET : TASK_SIZE_MAX) -#define STACK_TOP TASK_SIZE_LOW +#define MMAP_MAX (unsigned long)(test_thread_flag(TIF_ADDR32) ? \ + TASK_SIZE : TASK_SIZE_MAX) + +#define IBT_BITMAP_SIZE (round_up(MMAP_MAX, PAGE_SIZE * BITS_PER_BYTE) / \ + (PAGE_SIZE * BITS_PER_BYTE)) + +#define IBT_BITMAP_ADDR (TASK_SIZE - IBT_BITMAP_SIZE) + +#define STACK_TOP (TASK_SIZE_LOW < IBT_BITMAP_ADDR - PAGE_SIZE ? \ + TASK_SIZE_LOW : \ + IBT_BITMAP_ADDR - PAGE_SIZE) + #define STACK_TOP_MAX TASK_SIZE_MAX #define INIT_THREAD { \ diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 02243127dcf6..da39d4bde4e1 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,5 +20,6 @@ #define ARCH_X86_CET_ALLOC_SHSTK 0x3004 #define ARCH_X86_CET_GET_LEGACY_BITMAP 0x3005 /* deprecated */ #define ARCH_X86_CET_SET_LEGACY_BITMAP 0x3006 /* deprecated */ +#define ARCH_X86_CET_MARK_LEGACY_CODE 0x3007 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile index 311829335521..228906364513 100644 --- a/arch/x86/kernel/Makefile +++ b/arch/x86/kernel/Makefile @@ -140,7 +140,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o cet_bitmap.o ### # 64 bit specific files diff --git a/arch/x86/kernel/cet_bitmap.c b/arch/x86/kernel/cet_bitmap.c new file mode 100644 index 000000000000..25eb441eb094 --- /dev/null +++ b/arch/x86/kernel/cet_bitmap.c @@ -0,0 +1,210 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * For read fault, provide the zero page. For write fault coming from + * get_user_pages(), clear the page already allocated. + */ +static vm_fault_t bitmap_fault(const struct vm_special_mapping *sm, + struct vm_area_struct *vma, struct vm_fault *vmf) +{ + if (!(vmf->flags & FAULT_FLAG_WRITE)) { + vmf->page = ZERO_PAGE(vmf->address); + return 0; + } else { + vm_fault_t r; + + if (!vmf->cow_page) + return VM_FAULT_ERROR; + + clear_user_highpage(vmf->cow_page, vmf->address); + __SetPageUptodate(vmf->cow_page); + r = finish_fault(vmf); + return r ? r : VM_FAULT_DONE_COW; + } +} + +static int bitmap_mremap(const struct vm_special_mapping *sm, + struct vm_area_struct *vma) +{ + return -EINVAL; +} + +static const struct vm_special_mapping bitmap_mapping = { + .name = "[ibt_bitmap]", + .fault = bitmap_fault, + .mremap = bitmap_mremap, +}; + +static int alloc_bitmap(void) +{ + struct mm_struct *mm = current->mm; + struct vm_area_struct *vma; + u64 msr_ia32_u_cet; + int r = 0; + + if (down_write_killable(&mm->mmap_sem)) + return -EINTR; + + vma = _install_special_mapping(mm, IBT_BITMAP_ADDR, IBT_BITMAP_SIZE, + VM_READ | VM_MAYREAD | VM_MAYWRITE, + &bitmap_mapping); + + if (IS_ERR(vma)) + r = PTR_ERR(vma); + + up_write(&mm->mmap_sem); + + if (r) + return r; + + current->thread.cet.ibt_bitmap_used = 1; + + modify_fpu_regs_begin(); + rdmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); + msr_ia32_u_cet |= (MSR_IA32_CET_LEG_IW_EN | IBT_BITMAP_ADDR); + wrmsrl(MSR_IA32_U_CET, msr_ia32_u_cet); + modify_fpu_regs_end(); + return 0; +} + +/* + * Set bits in the IBT legacy code bitmap, which is read-only user memory. + */ +static int set_bits(unsigned long start_bit, unsigned long end_bit, + unsigned long set) +{ + unsigned long start_ul, end_ul, nr_ul; + unsigned long start_ul_addr, tmp_addr, len; + int i, j; + + start_ul = start_bit / BITS_PER_LONG; + end_ul = end_bit / BITS_PER_LONG; + i = start_bit % BITS_PER_LONG; + j = end_bit % BITS_PER_LONG; + + start_ul_addr = IBT_BITMAP_ADDR + start_ul * sizeof(0UL); + nr_ul = end_ul - start_ul + 1; + + tmp_addr = start_ul_addr; + len = nr_ul * sizeof(0UL); + + down_read(¤t->mm->mmap_sem); + while (len) { + unsigned long *first, *last, mask, bytes; + int ret, offset; + void *kern_page_addr; + struct page *page = NULL; + + ret = get_user_pages(tmp_addr, 1, FOLL_WRITE | FOLL_FORCE, + &page, NULL); + + if (ret <= 0) { + up_read(¤t->mm->mmap_sem); + return ret; + } + + kern_page_addr = kmap(page); + + bytes = len; + offset = tmp_addr & (PAGE_SIZE - 1); + + /* Is end_ul in this page? */ + if (bytes > (PAGE_SIZE - offset)) { + bytes = PAGE_SIZE - offset; + last = NULL; + } else { + last = (unsigned long *)(kern_page_addr + offset + bytes) - 1; + } + + /* Is start_ul in this page? */ + if (tmp_addr == start_ul_addr) + first = (unsigned long *)(kern_page_addr + offset); + else + first = NULL; + + if (nr_ul == 1) { + mask = GENMASK(j, i); + + if (set) + *first |= mask; + else + *first &= ~mask; + } else { + if (first) { + mask = GENMASK(BITS_PER_LONG - 1, i); + + if (set) + *first |= mask; + else + *first &= ~mask; + } + + if (last) { + mask = GENMASK(j, 0); + + if (set) + *last |= mask; + else + *last &= ~mask; + } + + if (nr_ul > 2) { + void *p = kern_page_addr + offset; + int cnt = bytes; + + if (first) { + p += sizeof(*first); + cnt -= sizeof(*first); + } + + if (last) + cnt -= sizeof(*last); + + if (set) + memset(p, 0xff, cnt); + else + memset(p, 0, cnt); + } + } + + set_page_dirty_lock(page); + kunmap(page); + put_page(page); + + len -= bytes; + tmp_addr += bytes; + } + up_read(¤t->mm->mmap_sem); + return 0; +} + +int cet_mark_legacy_code(unsigned long addr, unsigned long size, unsigned long set) +{ + int r; + + if (!current->thread.cet.ibt_enabled) + return -EINVAL; + + if ((addr >= IBT_BITMAP_ADDR) || (addr + size > IBT_BITMAP_ADDR)) + return -EINVAL; + + if (!current->thread.cet.ibt_bitmap_used) { + r = alloc_bitmap(); + if (r) + return r; + } + + return set_bits(addr / PAGE_SIZE, (addr + size - 1) / PAGE_SIZE, set); +} diff --git a/arch/x86/kernel/cet_prctl.c b/arch/x86/kernel/cet_prctl.c index 09d8c4ea935c..eec5baf8b0da 100644 --- a/arch/x86/kernel/cet_prctl.c +++ b/arch/x86/kernel/cet_prctl.c @@ -57,6 +57,18 @@ static int handle_alloc_shstk(unsigned long arg2) return 0; } +static int handle_mark_legacy_code(unsigned long arg2) +{ + unsigned long addr, size, set; + + if (get_user(addr, (unsigned long __user *)arg2) || + get_user(size, (unsigned long __user *)arg2 + 1) || + get_user(set, (unsigned long __user *)arg2 + 2)) + return -EFAULT; + + return cet_mark_legacy_code(addr, size, set); +} + int prctl_cet(int option, unsigned long arg2) { if (!cpu_x86_cet_enabled()) @@ -83,6 +95,9 @@ int prctl_cet(int option, unsigned long arg2) case ARCH_X86_CET_ALLOC_SHSTK: return handle_alloc_shstk(arg2); + case ARCH_X86_CET_MARK_LEGACY_CODE: + return handle_mark_legacy_code(arg2); + default: return -EINVAL; } diff --git a/mm/memory.c b/mm/memory.c index be93a73b5152..75076f727be0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3290,6 +3290,12 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, flush_icache_page(vma, page); entry = mk_pte(page, vma->vm_page_prot); + + if (is_zero_pfn(pte_pfn(entry))) { + entry = pte_mkspecial(entry); + goto alloc_set_pte_out; + } + if (write) entry = maybe_mkwrite(pte_mkdirty(entry), vma); /* copy-on-write page */ @@ -3302,6 +3308,8 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page)); page_add_file_rmap(page, false); } + +alloc_set_pte_out: set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); /* no need to invalidate: a not-present page won't be cached */