From patchwork Wed May 27 19:18:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573819 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B49B4912 for ; Wed, 27 May 2020 19:20:31 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 978D42089D for ; Wed, 27 May 2020 19:20:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 978D42089D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aC-00040U-0L; Wed, 27 May 2020 19:19:20 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aB-00040F-2C for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:19 +0000 X-Inumbo-ID: f27751dd-a04e-11ea-a777-12813bfff9fa Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f27751dd-a04e-11ea-a777-12813bfff9fa; Wed, 27 May 2020 19:19:14 +0000 (UTC) Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: pWuwntnoMuy1EWQAtQPoeE0QeTGmXZH8kVIvZIS1A0pYGGSDK0Iaqz1Wu/VZdWrDd2uD9kdgDp hXxKMHV/BedfVZ5aqPmMWJrOhAWvWRxeGpuIgBZSir7iyKh8oeMf2f0WaF/cGfrcAd7N/3glxy tHJ2omm/L1ade4G2eT5t3rvxHGzFaH7jl3Yvgkytrz4GdSqn3vxARB6zJz5kLjF+ySOcvcH6PH 8JlASs/wj6vHj7T0vaC/H2ehG67bvrm00Wlm2aFUQix7eZDi5IJggYT/mX8/I6Ed5a+f1YDXP+ KNs= X-SBRS: 2.7 X-MesageID: 18591104 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18591104" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 01/14] x86/traps: Clean up printing in {do_reserved, fatal}_trap() Date: Wed, 27 May 2020 20:18:34 +0100 Message-ID: <20200527191847.17207-2-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" For one, they render the vector in a different base. Introduce X86_EXC_* constants and vec_name() to refer to exceptions by their mnemonic, which starts bringing the code/diagnostics in line with the Intel and AMD manuals. Provide constants for every archtiecturally defined exception, even those not implemented by Xen yet, as do_reserved_trap() is a catch-all handler. Signed-off-by: Andrew Cooper Acked-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Move "#" into vec_name() to skip it for the 3-character vectors --- xen/arch/x86/traps.c | 26 +++++++++++++++++++++----- xen/include/asm-x86/processor.h | 6 +----- xen/include/asm-x86/x86-defns.h | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 10 deletions(-) diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index a8300c214d..427178e649 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -682,6 +682,22 @@ const char *trapstr(unsigned int trapnr) return trapnr < ARRAY_SIZE(strings) ? strings[trapnr] : "???"; } +static const char *vec_name(unsigned int vec) +{ + static const char names[][4] = { +#define P(x) [X86_EXC_ ## x] = "#" #x +#define N(x) [X86_EXC_ ## x] = #x + P(DE), P(DB), N(NMI), P(BP), P(OF), P(BR), P(UD), P(NM), + P(DF), N(CSO), P(TS), P(NP), P(SS), P(GP), P(PF), N(SPV), + P(MF), P(AC), P(MC), P(XM), P(VE), P(CP), + P(HV), P(VC), P(SX), +#undef N +#undef P + }; + + return (vec < ARRAY_SIZE(names) && names[vec][0]) ? names[vec] : "???"; +} + /* * This is called for faults at very unexpected times (e.g., when interrupts * are disabled). In such situations we can't do much that is safe. We try to @@ -739,10 +755,9 @@ void fatal_trap(const struct cpu_user_regs *regs, bool show_remote) } } - panic("FATAL TRAP: vector = %d (%s)\n" - "[error_code=%04x] %s\n", - trapnr, trapstr(trapnr), regs->error_code, - (regs->eflags & X86_EFLAGS_IF) ? "" : ", IN INTERRUPT CONTEXT"); + panic("FATAL TRAP: vec %u, %s[%04x]%s\n", + trapnr, vec_name(trapnr), regs->error_code, + (regs->eflags & X86_EFLAGS_IF) ? "" : " IN INTERRUPT CONTEXT"); } static void do_reserved_trap(struct cpu_user_regs *regs) @@ -753,7 +768,8 @@ static void do_reserved_trap(struct cpu_user_regs *regs) return; show_execution_state(regs); - panic("FATAL RESERVED TRAP %#x: %s\n", trapnr, trapstr(trapnr)); + panic("FATAL RESERVED TRAP: vec %u, %s[%04x]\n", + trapnr, vec_name(trapnr), regs->error_code); } static void do_trap(struct cpu_user_regs *regs) diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index 070691882b..96deac73ed 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -43,11 +43,7 @@ #define TRAP_virtualisation 20 #define TRAP_nr 32 -#define TRAP_HAVE_EC \ - ((1u << TRAP_double_fault) | (1u << TRAP_invalid_tss) | \ - (1u << TRAP_no_segment) | (1u << TRAP_stack_error) | \ - (1u << TRAP_gp_fault) | (1u << TRAP_page_fault) | \ - (1u << TRAP_alignment_check)) +#define TRAP_HAVE_EC X86_EXC_HAVE_EC /* Set for entry via SYSCALL. Informs return code to use SYSRETQ not IRETQ. */ /* NB. Same as VGCF_in_syscall. No bits in common with any other TRAP_ defn. */ diff --git a/xen/include/asm-x86/x86-defns.h b/xen/include/asm-x86/x86-defns.h index 8bf503220a..5366e2d018 100644 --- a/xen/include/asm-x86/x86-defns.h +++ b/xen/include/asm-x86/x86-defns.h @@ -118,4 +118,38 @@ #define X86_NR_VECTORS 256 +/* Exception Vectors */ +#define X86_EXC_DE 0 /* Divide Error. */ +#define X86_EXC_DB 1 /* Debug Exception. */ +#define X86_EXC_NMI 2 /* NMI. */ +#define X86_EXC_BP 3 /* Breakpoint. */ +#define X86_EXC_OF 4 /* Overflow. */ +#define X86_EXC_BR 5 /* BOUND Range. */ +#define X86_EXC_UD 6 /* Invalid Opcode. */ +#define X86_EXC_NM 7 /* Device Not Available. */ +#define X86_EXC_DF 8 /* Double Fault. */ +#define X86_EXC_CSO 9 /* Coprocessor Segment Overrun. */ +#define X86_EXC_TS 10 /* Invalid TSS. */ +#define X86_EXC_NP 11 /* Segment Not Present. */ +#define X86_EXC_SS 12 /* Stack-Segment Fault. */ +#define X86_EXC_GP 13 /* General Porection Fault. */ +#define X86_EXC_PF 14 /* Page Fault. */ +#define X86_EXC_SPV 15 /* PIC Spurious Interrupt Vector. */ +#define X86_EXC_MF 16 /* Maths fault (x87 FPU). */ +#define X86_EXC_AC 17 /* Alignment Check. */ +#define X86_EXC_MC 18 /* Machine Check. */ +#define X86_EXC_XM 19 /* SIMD Exception. */ +#define X86_EXC_VE 20 /* Virtualisation Exception. */ +#define X86_EXC_CP 21 /* Control-flow Protection. */ +#define X86_EXC_HV 28 /* Hypervisor Injection. */ +#define X86_EXC_VC 29 /* VMM Communication. */ +#define X86_EXC_SX 30 /* Security Exception. */ + +/* Bitmap of exceptions which have error codes. */ +#define X86_EXC_HAVE_EC \ + ((1u << X86_EXC_DF) | (1u << X86_EXC_TS) | (1u << X86_EXC_NP) | \ + (1u << X86_EXC_SS) | (1u << X86_EXC_GP) | (1u << X86_EXC_PF) | \ + (1u << X86_EXC_AC) | (1u << X86_EXC_CP) | \ + (1u << X86_EXC_VC) | (1u << X86_EXC_SX)) + #endif /* __XEN_X86_DEFNS_H__ */ From patchwork Wed May 27 19:18:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573811 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D91A4739 for ; Wed, 27 May 2020 19:20:16 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BF8DC207D3 for ; Wed, 27 May 2020 19:20:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF8DC207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1a7-0003zq-FO; Wed, 27 May 2020 19:19:15 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1a6-0003zb-5F for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:14 +0000 X-Inumbo-ID: f27751dc-a04e-11ea-a777-12813bfff9fa Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f27751dc-a04e-11ea-a777-12813bfff9fa; Wed, 27 May 2020 19:19:12 +0000 (UTC) Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: ija9DmiPgoPmZNcL3bjjdKcpLaVY+8guYONj0oMriHZeiKRsozlQ4o9aitlckvP1s6btqJjxh1 a18qRYR3chz2xqEmZi1iYjPkG72SPuWjfXyhVFoGpAviU1XOrbI66ZIQIbfdo4gTDSN5jj9AgB s6lRv7mYcHllksiQ687H74WJMWuU5rfZhBYRoEdw8kjtIZt8gT6+KHdasj18k6r8n/CwzOQovE kl3xiEYvA2VvsgwwvuZvjliwLI/IDNpuSXuy1dJz9R6NEaTju6WmHDxpgad7sSTQhi6ISl/cjt uPA= X-SBRS: 2.7 X-MesageID: 18591103 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18591103" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 02/14] x86/traps: Factor out extable_fixup() and make printing consistent Date: Wed, 27 May 2020 20:18:35 +0100 Message-ID: <20200527191847.17207-3-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" UD faults never had any diagnostics printed, and the others were inconsistent. Don't use dprintk() because identifying traps.c is actively unhelpful in the message, as it is the location of the fixup, not the fault. Use the new vec_name() infrastructure, rather than leaving raw numbers for the log. (XEN) Running stub recovery selftests... (XEN) Fixup #UD[0000]: ffff82d07fffd040 [ffff82d07fffd040] -> ffff82d0403ac9d6 (XEN) Fixup #GP[0000]: ffff82d07fffd041 [ffff82d07fffd041] -> ffff82d0403ac9d6 (XEN) Fixup #SS[0000]: ffff82d07fffd040 [ffff82d07fffd040] -> ffff82d0403ac9d6 (XEN) Fixup #BP[0000]: ffff82d07fffd041 [ffff82d07fffd041] -> ffff82d0403ac9d6 Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Rebase * Rename to extable_fixup() to better distinguish from fixup_page_fault() --- xen/arch/x86/traps.c | 77 ++++++++++++++++++++++++---------------------------- 1 file changed, 35 insertions(+), 42 deletions(-) diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 427178e649..eeb3e146ef 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -772,10 +772,31 @@ static void do_reserved_trap(struct cpu_user_regs *regs) trapnr, vec_name(trapnr), regs->error_code); } +static bool extable_fixup(struct cpu_user_regs *regs, bool print) +{ + unsigned long fixup = search_exception_table(regs); + + if ( unlikely(fixup == 0) ) + return false; + + /* + * Don't use dprintk() because the __FILE__ reference is unhelpful. + * Can currently be triggered by guests. Make sure we ratelimit. + */ + if ( IS_ENABLED(CONFIG_DEBUG) && print ) + printk(XENLOG_GUEST XENLOG_WARNING "Fixup %s[%04x]: %p [%ps] -> %p\n", + vec_name(regs->entry_vector), regs->error_code, + _p(regs->rip), _p(regs->rip), _p(fixup)); + + regs->rip = fixup; + this_cpu(last_extable_addr) = regs->rip; + + return true; +} + static void do_trap(struct cpu_user_regs *regs) { unsigned int trapnr = regs->entry_vector; - unsigned long fixup; if ( regs->error_code & X86_XEC_EXT ) goto hardware_trap; @@ -793,14 +814,8 @@ static void do_trap(struct cpu_user_regs *regs) return; } - if ( likely((fixup = search_exception_table(regs)) != 0) ) - { - dprintk(XENLOG_ERR, "Trap %u: %p [%ps] -> %p\n", - trapnr, _p(regs->rip), _p(regs->rip), _p(fixup)); - this_cpu(last_extable_addr) = regs->rip; - regs->rip = fixup; + if ( likely(extable_fixup(regs, true)) ) return; - } hardware_trap: if ( debugger_trap_fatal(trapnr, regs) ) @@ -1108,12 +1123,8 @@ void do_invalid_op(struct cpu_user_regs *regs) } die: - if ( (fixup = search_exception_table(regs)) != 0 ) - { - this_cpu(last_extable_addr) = regs->rip; - regs->rip = fixup; + if ( likely(extable_fixup(regs, true)) ) return; - } if ( debugger_trap_fatal(TRAP_invalid_op, regs) ) return; @@ -1129,16 +1140,8 @@ void do_int3(struct cpu_user_regs *regs) if ( !guest_mode(regs) ) { - unsigned long fixup; - - if ( (fixup = search_exception_table(regs)) != 0 ) - { - this_cpu(last_extable_addr) = regs->rip; - dprintk(XENLOG_DEBUG, "Trap %u: %p [%ps] -> %p\n", - TRAP_int3, _p(regs->rip), _p(regs->rip), _p(fixup)); - regs->rip = fixup; + if ( likely(extable_fixup(regs, true)) ) return; - } if ( !debugger_trap_fatal(TRAP_int3, regs) ) printk(XENLOG_DEBUG "Hit embedded breakpoint at %p [%ps]\n", @@ -1415,7 +1418,7 @@ static int fixup_page_fault(unsigned long addr, struct cpu_user_regs *regs) void do_page_fault(struct cpu_user_regs *regs) { - unsigned long addr, fixup; + unsigned long addr; unsigned int error_code; addr = read_cr2(); @@ -1461,11 +1464,9 @@ void do_page_fault(struct cpu_user_regs *regs) if ( pf_type != real_fault ) return; - if ( likely((fixup = search_exception_table(regs)) != 0) ) + if ( likely(extable_fixup(regs, false)) ) { perfc_incr(copy_user_faults); - this_cpu(last_extable_addr) = regs->rip; - regs->rip = fixup; return; } @@ -1521,7 +1522,6 @@ void do_general_protection(struct cpu_user_regs *regs) #ifdef CONFIG_PV struct vcpu *v = current; #endif - unsigned long fixup; if ( debugger_trap_entry(TRAP_gp_fault, regs) ) return; @@ -1588,14 +1588,8 @@ void do_general_protection(struct cpu_user_regs *regs) gp_in_kernel: - if ( likely((fixup = search_exception_table(regs)) != 0) ) - { - dprintk(XENLOG_INFO, "GPF (%04x): %p [%ps] -> %p\n", - regs->error_code, _p(regs->rip), _p(regs->rip), _p(fixup)); - this_cpu(last_extable_addr) = regs->rip; - regs->rip = fixup; + if ( likely(extable_fixup(regs, true)) ) return; - } hardware_gp: if ( debugger_trap_fatal(TRAP_gp_fault, regs) ) @@ -1754,18 +1748,17 @@ void do_device_not_available(struct cpu_user_regs *regs) if ( !guest_mode(regs) ) { - unsigned long fixup = search_exception_table(regs); - - gprintk(XENLOG_ERR, "#NM: %p [%ps] -> %p\n", - _p(regs->rip), _p(regs->rip), _p(fixup)); /* * We shouldn't be able to reach here, but for release builds have * the recovery logic in place nevertheless. */ - ASSERT_UNREACHABLE(); - BUG_ON(!fixup); - regs->rip = fixup; - return; + if ( extable_fixup(regs, true) ) + { + ASSERT_UNREACHABLE(); + return; + } + + fatal_trap(regs, false); } #ifdef CONFIG_PV From patchwork Wed May 27 19:18:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573809 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ACE73739 for ; Wed, 27 May 2020 19:20:13 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 93718207D3 for ; Wed, 27 May 2020 19:20:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93718207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aB-00040L-Nh; Wed, 27 May 2020 19:19:19 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aA-000407-PL for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:18 +0000 X-Inumbo-ID: f267606a-a04e-11ea-81bc-bc764e2007e4 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f267606a-a04e-11ea-81bc-bc764e2007e4; Wed, 27 May 2020 19:19:12 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: PqU4pUb91tB9Qzz3QFZ5kdCgJyyeaIaQRM6P1RfR9bMkxLxj5+zFmOGBmAEaXDS6T9LmWcaiR/ NznYQ7Ek1SAKgop8OoJXAsDup0hcFx2/qh5n7dSxICCbcOWKmVwtWImqvP+CbZs1q14aVZX8hl 5omvACdCjxk6j1FDGP6amTVLSx6ufQyG5scC+Xns65sJgWJ+Ioj9Zq51LxH+cMQMtBsKpW0w9x s+VKO+5C8cJvfrTjrB055dJH8Po8Js3Dq2IPx3RxeeffSWkFxxl2MViGpCKNFEGgI3v39k2QT0 Dgw= X-SBRS: 2.7 X-MesageID: 19333916 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="19333916" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 03/14] x86/shstk: Introduce Supervisor Shadow Stack support Date: Wed, 27 May 2020 20:18:36 +0100 Message-ID: <20200527191847.17207-4-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Introduce CONFIG_HAS_AS_CET to determine whether CET instructions are supported in the assembler, and CONFIG_XEN_SHSTK as the main build option. Introduce cet={no-,}shstk to for a user to select whether or not to use shadow stacks at runtime, and X86_FEATURE_XEN_SHSTK to determine Xen's overall enablement of shadow stacks. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné LLVM 6 supports CET-SS instructions while only LLVM 7 supports CET-IBT instructions. We'd need to split HAS_AS_CET into two if we want to support supervisor shadow stacks with LLVM 6. (This demonstrates exactly why picking a handful of instructions to test is the right approach.) v2: * Leave a comment identifying minimum toolchain support, to make it easier to remove ifdefary in the future when bumping minima. * Reindent CONFIG_XEN_SHSTK help text. * Rename xen= to cet=. Add documentation, __init. --- docs/misc/xen-command-line.pandoc | 17 +++++++++++++++++ xen/arch/x86/Kconfig | 18 ++++++++++++++++++ xen/arch/x86/setup.c | 30 ++++++++++++++++++++++++++++++ xen/include/asm-x86/cpufeature.h | 1 + xen/include/asm-x86/cpufeatures.h | 1 + xen/scripts/Kconfig.include | 4 ++++ 6 files changed, 71 insertions(+) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index e16bb90184..d4934eabb7 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -270,6 +270,23 @@ and not running softirqs. Reduce this if softirqs are not being run frequently enough. Setting this to a high value may cause boot failure, particularly if the NMI watchdog is also enabled. +### cet + = List of [ shstk= ] + + Applicability: x86 + +Controls for the use of Control-flow Enforcement Technology. CET is group of +hardware features designed to combat Return-oriented Programming (ROP, also +call/jmp COP/JOP) attacks. + +* The `shstk=` boolean controls whether Xen uses Shadow Stacks for its own + protection. + + The option is available when `CONFIG_XEN_SHSTK` is compiled in, and + defaults to `true` on hardware supporting CET-SS. Specifying + `cet=no-shstk` will cause Xen not to use Shadow Stacks even when support + is available in hardware. + ### clocksource (x86) > `= pit | hpet | acpi | tsc` diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig index b565f6831d..304a42ffb2 100644 --- a/xen/arch/x86/Kconfig +++ b/xen/arch/x86/Kconfig @@ -34,6 +34,10 @@ config ARCH_DEFCONFIG config INDIRECT_THUNK def_bool $(cc-option,-mindirect-branch-register) +config HAS_AS_CET + # binutils >= 2.29 and LLVM >= 7 + def_bool $(as-instr,wrssq %rax$(comma)0;setssbsy;endbr64) + menu "Architecture Features" source "arch/Kconfig" @@ -97,6 +101,20 @@ config HVM If unsure, say Y. +config XEN_SHSTK + bool "Supervisor Shadow Stacks" + depends on HAS_AS_CET && EXPERT = "y" + default y + ---help--- + Control-flow Enforcement Technology (CET) is a set of features in + hardware designed to combat Return-oriented Programming (ROP, also + call/jump COP/JOP) attacks. Shadow Stacks are one CET feature + designed to provide return address protection. + + This option arranges for Xen to use CET-SS for its own protection. + When CET-SS is active, 32bit PV guests cannot be used. Backwards + compatiblity can be provided vai the PV Shim mechanism. + config SHADOW_PAGING bool "Shadow Paging" default y diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 2dec7a3fc6..584589baff 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -95,6 +95,36 @@ unsigned long __initdata highmem_start; size_param("highmem-start", highmem_start); #endif +static bool __initdata opt_xen_shstk = true; + +static int __init parse_cet(const char *s) +{ + const char *ss; + int val, rc = 0; + + do { + ss = strchr(s, ','); + if ( !ss ) + ss = strchr(s, '\0'); + + if ( (val = parse_boolean("shstk", s, ss)) >= 0 ) + { +#ifdef CONFIG_XEN_SHSTK + opt_xen_shstk = val; +#else + no_config_param("XEN_SHSTK", "cet", s, ss); +#endif + } + else + rc = -EINVAL; + + s = ss + 1; + } while ( *ss ); + + return rc; +} +custom_param("cet", parse_cet); + cpumask_t __read_mostly cpu_present_map; unsigned long __read_mostly xen_phys_start; diff --git a/xen/include/asm-x86/cpufeature.h b/xen/include/asm-x86/cpufeature.h index cadef4e824..b831448eba 100644 --- a/xen/include/asm-x86/cpufeature.h +++ b/xen/include/asm-x86/cpufeature.h @@ -137,6 +137,7 @@ #define cpu_has_aperfmperf boot_cpu_has(X86_FEATURE_APERFMPERF) #define cpu_has_lfence_dispatch boot_cpu_has(X86_FEATURE_LFENCE_DISPATCH) #define cpu_has_xen_lbr boot_cpu_has(X86_FEATURE_XEN_LBR) +#define cpu_has_xen_shstk boot_cpu_has(X86_FEATURE_XEN_SHSTK) #define cpu_has_msr_tsc_aux (cpu_has_rdtscp || cpu_has_rdpid) diff --git a/xen/include/asm-x86/cpufeatures.h b/xen/include/asm-x86/cpufeatures.h index b9d3cac975..d7e42d9bb6 100644 --- a/xen/include/asm-x86/cpufeatures.h +++ b/xen/include/asm-x86/cpufeatures.h @@ -38,6 +38,7 @@ XEN_CPUFEATURE(XEN_LBR, X86_SYNTH(22)) /* Xen uses MSR_DEBUGCTL.LBR */ XEN_CPUFEATURE(SC_VERW_PV, X86_SYNTH(23)) /* VERW used by Xen for PV */ XEN_CPUFEATURE(SC_VERW_HVM, X86_SYNTH(24)) /* VERW used by Xen for HVM */ XEN_CPUFEATURE(SC_VERW_IDLE, X86_SYNTH(25)) /* VERW used by Xen for idle */ +XEN_CPUFEATURE(XEN_SHSTK, X86_SYNTH(26)) /* Xen uses CET Shadow Stacks */ /* Bug words follow the synthetic words. */ #define X86_NR_BUG 1 diff --git a/xen/scripts/Kconfig.include b/xen/scripts/Kconfig.include index 8221095ca3..e1f13e1720 100644 --- a/xen/scripts/Kconfig.include +++ b/xen/scripts/Kconfig.include @@ -31,6 +31,10 @@ cc-option = $(success,$(CC) -Werror $(CLANG_FLAGS) $(1) -E -x c /dev/null -o /de # Return y if the linker supports , n otherwise ld-option = $(success,$(LD) -v $(1)) +# $(as-instr,) +# Return y if the assembler supports , n otherwise +as-instr = $(success,printf "%b\n" "$(1)" | $(CC) $(CLANG_FLAGS) -c -x assembler -o /dev/null -) + # check if $(CC) and $(LD) exist $(error-if,$(failure,command -v $(CC)),compiler '$(CC)' not found) $(error-if,$(failure,command -v $(LD)),linker '$(LD)' not found) From patchwork Wed May 27 19:18:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573817 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 007D1739 for ; Wed, 27 May 2020 19:20:23 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA073207D3 for ; Wed, 27 May 2020 19:20:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA073207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aM-00045h-BX; Wed, 27 May 2020 19:19:30 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aL-00044e-2t for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:29 +0000 X-Inumbo-ID: f518fc1b-a04e-11ea-a777-12813bfff9fa Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f518fc1b-a04e-11ea-a777-12813bfff9fa; Wed, 27 May 2020 19:19:18 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: KPMzHN7M4o1atLAbnD8TJWcgtq/9W44k99HeJc51xZ7vy4wcR5MQtaZJzm2CQRb+Ipd249maSA OaNFGgLYbqehzstG49rMtAVS/JzuM0k0ReG9EOH6uo6/HzEi9Q8++2XrVFWwYIBnNTRi4KllVz +4x4wdu5Lz5xRs2n6pYNOQnrFVW7ZHQqnGxgvwKAmi+0TgLBB6AYYvhTZcTWG1za+cpJP4B3Gm vSFvFtaX+5p/yZLjM28gA1QiJ/4pKYlUDpjvMmSozDeI4tHq/hkXy477SMg3vBDaRUdJZ32oMY 9Co= X-SBRS: 2.7 X-MesageID: 19333923 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="19333923" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 04/14] x86/traps: Implement #CP handler and extend #PF for shadow stacks Date: Wed, 27 May 2020 20:18:37 +0100 Message-ID: <20200527191847.17207-5-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" For now, any #CP exception or shadow stack #PF indicate a bug in Xen, but attempt to recover from #CP if taken in guest context. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Rebase over #PF[Rsvd] rework. * Alignment for PFEC_shstk. * Use more X86_EXC_* names. --- xen/arch/x86/traps.c | 46 +++++++++++++++++++++++++++++++++++++++-- xen/arch/x86/x86_64/entry.S | 7 ++++++- xen/include/asm-x86/processor.h | 2 ++ 3 files changed, 52 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index eeb3e146ef..90da787ee2 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -156,7 +156,9 @@ void (* const exception_table[TRAP_nr])(struct cpu_user_regs *regs) = { [TRAP_alignment_check] = do_trap, [TRAP_machine_check] = (void *)do_machine_check, [TRAP_simd_error] = do_trap, - [TRAP_virtualisation ... + [TRAP_virtualisation] = do_reserved_trap, + [X86_EXC_CP] = do_entry_CP, + [X86_EXC_CP + 1 ... (ARRAY_SIZE(exception_table) - 1)] = do_reserved_trap, }; @@ -1445,8 +1447,10 @@ void do_page_fault(struct cpu_user_regs *regs) * * Anything remaining is an error, constituting corruption of the * pagetables and probably an L1TF vulnerable gadget. + * + * Any shadow stack access fault is a bug in Xen. */ - if ( error_code & PFEC_reserved_bit ) + if ( error_code & (PFEC_reserved_bit | PFEC_shstk) ) goto fatal; if ( unlikely(!guest_mode(regs)) ) @@ -1898,6 +1902,43 @@ void do_debug(struct cpu_user_regs *regs) pv_inject_hw_exception(TRAP_debug, X86_EVENT_NO_EC); } +void do_entry_CP(struct cpu_user_regs *regs) +{ + static const char errors[][10] = { + [1] = "near ret", + [2] = "far/iret", + [3] = "endbranch", + [4] = "rstorssp", + [5] = "setssbsy", + }; + const char *err = "??"; + unsigned int ec = regs->error_code; + + if ( debugger_trap_entry(TRAP_debug, regs) ) + return; + + /* Decode ec if possible */ + if ( ec < ARRAY_SIZE(errors) && errors[ec][0] ) + err = errors[ec]; + + /* + * For now, only supervisors shadow stacks should be active. A #CP from + * guest context is probably a Xen bug, but kill the guest in an attempt + * to recover. + */ + if ( guest_mode(regs) ) + { + gprintk(XENLOG_ERR, "Hit #CP[%04x] in guest context %04x:%p\n", + ec, regs->cs, _p(regs->rip)); + ASSERT_UNREACHABLE(); + domain_crash(current->domain); + return; + } + + show_execution_state(regs); + panic("CONTROL-FLOW PROTECTION FAULT: #CP[%04x] %s\n", ec, err); +} + static void __init noinline __set_intr_gate(unsigned int n, uint32_t dpl, void *addr) { @@ -1987,6 +2028,7 @@ void __init init_idt_traps(void) set_intr_gate(TRAP_alignment_check,&alignment_check); set_intr_gate(TRAP_machine_check,&machine_check); set_intr_gate(TRAP_simd_error,&simd_coprocessor_error); + set_intr_gate(X86_EXC_CP, entry_CP); /* Specify dedicated interrupt stacks for NMI, #DF, and #MC. */ enable_each_ist(idt_table); diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S index d55453f3f3..f7ee3dce91 100644 --- a/xen/arch/x86/x86_64/entry.S +++ b/xen/arch/x86/x86_64/entry.S @@ -795,6 +795,10 @@ ENTRY(alignment_check) movl $TRAP_alignment_check,4(%rsp) jmp handle_exception +ENTRY(entry_CP) + movl $X86_EXC_CP, 4(%rsp) + jmp handle_exception + ENTRY(double_fault) movl $TRAP_double_fault,4(%rsp) /* Set AC to reduce chance of further SMAP faults */ @@ -940,7 +944,8 @@ autogen_stubs: /* Automatically generated stubs. */ entrypoint 1b /* Reserved exceptions, heading towards do_reserved_trap(). */ - .elseif vec == TRAP_copro_seg || vec == TRAP_spurious_int || (vec > TRAP_simd_error && vec < TRAP_nr) + .elseif vec == X86_EXC_CSO || vec == X86_EXC_SPV || \ + vec == X86_EXC_VE || (vec > X86_EXC_CP && vec < TRAP_nr) 1: test $8,%spl /* 64bit exception frames are 16 byte aligned, but the word */ jz 2f /* size is 8 bytes. Check whether the processor gave us an */ diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index 96deac73ed..c2b9dc1ac0 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -68,6 +68,7 @@ #define PFEC_reserved_bit (_AC(1,U) << 3) #define PFEC_insn_fetch (_AC(1,U) << 4) #define PFEC_prot_key (_AC(1,U) << 5) +#define PFEC_shstk (_AC(1,U) << 6) #define PFEC_arch_mask (_AC(0xffff,U)) /* Architectural PFEC values. */ /* Internally used only flags. */ #define PFEC_page_paged (1U<<16) @@ -530,6 +531,7 @@ DECLARE_TRAP_HANDLER(coprocessor_error); DECLARE_TRAP_HANDLER(simd_coprocessor_error); DECLARE_TRAP_HANDLER_CONST(machine_check); DECLARE_TRAP_HANDLER(alignment_check); +DECLARE_TRAP_HANDLER(entry_CP); DECLARE_TRAP_HANDLER(entry_int82); From patchwork Wed May 27 19:18:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573813 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E216E912 for ; Wed, 27 May 2020 19:20:17 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C8C04207D3 for ; Wed, 27 May 2020 19:20:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C8C04207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aM-00045B-1O; Wed, 27 May 2020 19:19:30 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aK-00044V-Pr for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:28 +0000 X-Inumbo-ID: f35ede3a-a04e-11ea-81bc-bc764e2007e4 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f35ede3a-a04e-11ea-81bc-bc764e2007e4; Wed, 27 May 2020 19:19:14 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 6yVJg/lifPGO70PS2wgpoTRI1lbe6ilhKsuQKwn77mP8iPyQN+41b9d9G5++vpGzQ/ftQCiYYd ZgK+D1g3IJgzBulmn6XKKAh8lQHXEAR013SdwW9z7Es8GkxZwhq9lNVQL5o9+mJd8UZr4qw70k DPVef143ILoMh4hgxea53PUJo+MtnUWnGFsATZcYwMX6MwBk9SgtFF0lOnRrv5Cqp/CgWoewgF didimL59PEPr413Qm8eYsWKX+P+TE+UcqnnH6LksOZJyea05myySxgXXiSk46AV0q/4CMu/vFY 1Qo= X-SBRS: 2.7 X-MesageID: 19333917 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="19333917" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 05/14] x86/shstk: Re-layout the stack block for shadow stacks Date: Wed, 27 May 2020 20:18:38 +0100 Message-ID: <20200527191847.17207-6-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We have two free pages in the current stack. A useful property of shadow stacks and regular stacks is that they act as each others guard pages as far as OoB writes go. Move the regular IST stacks up by one page, to allow their shadow stack page to be in slot 0. The primary shadow stack uses slot 5. As the shadow IST stacks are only 1k large, shuffle the order of IST vectors to have #DF numerically highest (so there is no chance of a shadow stack overflow clobbering the supervisor token). The XPTI code already breaks the MEMORY_GUARD abstraction for stacks by forcing it to be present. To avoid having too many configurations, do away with the concept entirely, and unconditionally unmap the pages in all cases. A later change will turn these properly into shadow stacks. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Adjust text * Introduce PRIMARY_SHSTK_SLOT (Name subject to improvement). --- xen/arch/x86/cpu/common.c | 10 +++++----- xen/arch/x86/mm.c | 19 ++++++------------- xen/arch/x86/smpboot.c | 3 +-- xen/arch/x86/traps.c | 23 ++++++----------------- xen/include/asm-x86/config.h | 3 +++ xen/include/asm-x86/current.h | 12 ++++++------ xen/include/asm-x86/mm.h | 1 - xen/include/asm-x86/processor.h | 6 +++--- 8 files changed, 30 insertions(+), 47 deletions(-) diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index 09b911b3ba..690fd8baa8 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -753,14 +753,14 @@ void load_system_tables(void) * valid on every instruction boundary. (Note: these are all * semantically ACCESS_ONCE() due to tss's volatile qualifier.) * - * rsp0 refers to the primary stack. #MC, #DF, NMI and #DB handlers + * rsp0 refers to the primary stack. #MC, NMI, #DB and #DF handlers * each get their own stacks. No IO Bitmap. */ tss->rsp0 = stack_bottom; - tss->ist[IST_MCE - 1] = stack_top + IST_MCE * PAGE_SIZE; - tss->ist[IST_DF - 1] = stack_top + IST_DF * PAGE_SIZE; - tss->ist[IST_NMI - 1] = stack_top + IST_NMI * PAGE_SIZE; - tss->ist[IST_DB - 1] = stack_top + IST_DB * PAGE_SIZE; + tss->ist[IST_MCE - 1] = stack_top + (1 + IST_MCE) * PAGE_SIZE; + tss->ist[IST_NMI - 1] = stack_top + (1 + IST_NMI) * PAGE_SIZE; + tss->ist[IST_DB - 1] = stack_top + (1 + IST_DB) * PAGE_SIZE; + tss->ist[IST_DF - 1] = stack_top + (1 + IST_DF) * PAGE_SIZE; tss->bitmap = IOBMP_INVALID_OFFSET; /* All other stack pointers poisioned. */ diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index e42044eb74..2f1e716b6d 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -5996,25 +5996,18 @@ void memguard_unguard_range(void *p, unsigned long l) void memguard_guard_stack(void *p) { - /* IST_MAX IST pages + at least 1 guard page + primary stack. */ - BUILD_BUG_ON((IST_MAX + 1) * PAGE_SIZE + PRIMARY_STACK_SIZE > STACK_SIZE); + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, _PAGE_NONE); - memguard_guard_range(p + IST_MAX * PAGE_SIZE, - STACK_SIZE - PRIMARY_STACK_SIZE - IST_MAX * PAGE_SIZE); + p += PRIMARY_SHSTK_SLOT * PAGE_SIZE; + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, _PAGE_NONE); } void memguard_unguard_stack(void *p) { - memguard_unguard_range(p + IST_MAX * PAGE_SIZE, - STACK_SIZE - PRIMARY_STACK_SIZE - IST_MAX * PAGE_SIZE); -} - -bool memguard_is_stack_guard_page(unsigned long addr) -{ - addr &= STACK_SIZE - 1; + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, PAGE_HYPERVISOR_RW); - return addr >= IST_MAX * PAGE_SIZE && - addr < STACK_SIZE - PRIMARY_STACK_SIZE; + p += PRIMARY_SHSTK_SLOT * PAGE_SIZE; + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, PAGE_HYPERVISOR_RW); } void arch_dump_shared_mem_info(void) diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c index 170ab24e66..13b3dade9c 100644 --- a/xen/arch/x86/smpboot.c +++ b/xen/arch/x86/smpboot.c @@ -823,8 +823,7 @@ static int setup_cpu_root_pgt(unsigned int cpu) /* Install direct map page table entries for stack, IDT, and TSS. */ for ( off = rc = 0; !rc && off < STACK_SIZE; off += PAGE_SIZE ) - if ( !memguard_is_stack_guard_page(off) ) - rc = clone_mapping(__va(__pa(stack_base[cpu])) + off, rpt); + rc = clone_mapping(__va(__pa(stack_base[cpu])) + off, rpt); if ( !rc ) rc = clone_mapping(idt_tables[cpu], rpt); diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 90da787ee2..235a72cf4a 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -365,20 +365,15 @@ static void show_guest_stack(struct vcpu *v, const struct cpu_user_regs *regs) /* * Notes for get_stack_trace_bottom() and get_stack_dump_bottom() * - * Stack pages 0 - 3: + * Stack pages 1 - 4: * These are all 1-page IST stacks. Each of these stacks have an exception * frame and saved register state at the top. The interesting bound for a * trace is the word adjacent to this, while the bound for a dump is the * very top, including the exception frame. * - * Stack pages 4 and 5: - * None of these are particularly interesting. With MEMORY_GUARD, page 5 is - * explicitly not present, so attempting to dump or trace it is - * counterproductive. Without MEMORY_GUARD, it is possible for a call chain - * to use the entire primary stack and wander into page 5. In this case, - * consider these pages an extension of the primary stack to aid debugging - * hopefully rare situations where the primary stack has effective been - * overflown. + * Stack pages 0 and 5: + * Shadow stacks. These are mapped read-only, and used by CET-SS capable + * processors. They will never contain regular stack data. * * Stack pages 6 and 7: * These form the primary stack, and have a cpu_info at the top. For a @@ -392,13 +387,10 @@ unsigned long get_stack_trace_bottom(unsigned long sp) { switch ( get_stack_page(sp) ) { - case 0 ... 3: + case 1 ... 4: return ROUNDUP(sp, PAGE_SIZE) - offsetof(struct cpu_user_regs, es) - sizeof(unsigned long); -#ifndef MEMORY_GUARD - case 4 ... 5: -#endif case 6 ... 7: return ROUNDUP(sp, STACK_SIZE) - sizeof(struct cpu_info) - sizeof(unsigned long); @@ -412,12 +404,9 @@ unsigned long get_stack_dump_bottom(unsigned long sp) { switch ( get_stack_page(sp) ) { - case 0 ... 3: + case 1 ... 4: return ROUNDUP(sp, PAGE_SIZE) - sizeof(unsigned long); -#ifndef MEMORY_GUARD - case 4 ... 5: -#endif case 6 ... 7: return ROUNDUP(sp, STACK_SIZE) - sizeof(unsigned long); diff --git a/xen/include/asm-x86/config.h b/xen/include/asm-x86/config.h index 266d281718..f3cf5df462 100644 --- a/xen/include/asm-x86/config.h +++ b/xen/include/asm-x86/config.h @@ -75,6 +75,9 @@ /* Primary stack is restricted to 8kB by guard pages. */ #define PRIMARY_STACK_SIZE 8192 +/* Primary shadow stack is slot 5 of 8, immediately under the primary stack. */ +#define PRIMARY_SHSTK_SLOT 5 + /* Total size of syscall and emulation stubs. */ #define STUB_BUF_SHIFT (L1_CACHE_SHIFT > 7 ? L1_CACHE_SHIFT : 7) #define STUB_BUF_SIZE (1 << STUB_BUF_SHIFT) diff --git a/xen/include/asm-x86/current.h b/xen/include/asm-x86/current.h index 5b8f4dbc79..99b66a0087 100644 --- a/xen/include/asm-x86/current.h +++ b/xen/include/asm-x86/current.h @@ -16,12 +16,12 @@ * * 7 - Primary stack (with a struct cpu_info at the top) * 6 - Primary stack - * 5 - Optionally not present (MEMORY_GUARD) - * 4 - Unused; optionally not present (MEMORY_GUARD) - * 3 - Unused; optionally not present (MEMORY_GUARD) - * 2 - MCE IST stack - * 1 - NMI IST stack - * 0 - Double Fault IST stack + * 5 - Primay Shadow Stack (read-only) + * 4 - #DF IST stack + * 3 - #DB IST stack + * 2 - NMI IST stack + * 1 - #MC IST stack + * 0 - IST Shadow Stacks (4x 1k, read-only) */ /* diff --git a/xen/include/asm-x86/mm.h b/xen/include/asm-x86/mm.h index 3d3f9d49ac..7e74996053 100644 --- a/xen/include/asm-x86/mm.h +++ b/xen/include/asm-x86/mm.h @@ -536,7 +536,6 @@ void memguard_unguard_range(void *p, unsigned long l); void memguard_guard_stack(void *p); void memguard_unguard_stack(void *p); -bool __attribute_const__ memguard_is_stack_guard_page(unsigned long addr); struct mmio_ro_emulate_ctxt { unsigned long cr2; diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index c2b9dc1ac0..8ab09cf7ed 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -440,10 +440,10 @@ struct tss_page { DECLARE_PER_CPU(struct tss_page, tss_page); #define IST_NONE 0UL -#define IST_DF 1UL +#define IST_MCE 1UL #define IST_NMI 2UL -#define IST_MCE 3UL -#define IST_DB 4UL +#define IST_DB 3UL +#define IST_DF 4UL #define IST_MAX 4UL /* Set the Interrupt Stack Table used by a particular IDT entry. */ From patchwork Wed May 27 19:18:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3DF6F739 for ; Wed, 27 May 2020 19:21:06 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 227F3207D3 for ; Wed, 27 May 2020 19:21:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 227F3207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aW-0004EE-4V; Wed, 27 May 2020 19:19:40 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aU-0004Cq-Pr for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:38 +0000 X-Inumbo-ID: f5699b3e-a04e-11ea-9947-bc764e2007e4 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f5699b3e-a04e-11ea-9947-bc764e2007e4; Wed, 27 May 2020 19:19:17 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 3iRO9kwgKCNdvZIluTTdpAT/32WruaeuHu2KpnABpZyIvZ+Vr+VV4xpohS0xAvZrALgMKsTVd6 tWXKr9ZjuXo8i8o3L8YyYuI0aquhbf2e7yM0Kw8dJUXJ7pt/sjFnQc0aH57p3BPAFYn/aHyq1G UPsZHVhzGAOR0+euvfSZAcCq7Or4hRjqb5HLesPc3oa/nqlHVoEtsOQU2lUYxXoZ4vGSdoAo4w //HwxzzZoOzhP67f1BplRf2QTfz9snO/vK1N00DpAP4VYboMCNwaeDFS7Y3yVNwxzNMo+nm0bi 9d4= X-SBRS: 2.7 X-MesageID: 18946803 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18946803" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 06/14] x86/shstk: Create shadow stacks Date: Wed, 27 May 2020 20:18:39 +0100 Message-ID: <20200527191847.17207-7-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Introduce HYPERVISOR_SHSTK pagetable constants, which are Read-Only + Dirty. Use these in place of _PAGE_NONE for memguard_guard_stack(). Supervisor shadow stacks need a token written at the top, which is most easily done before making the frame read only. Allocate the shadow IST stack block in struct tss_page. It doesn't strictly need to live here, but it is a convenient location (and XPTI-safe, for testing purposes), and placing it ahead of the TSS doesn't risk colliding with a bad IO Bitmap offset and turning into some IO port permissions. Have load_system_tables() set up the shadow IST stack table when setting up the regular IST in the TSS. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Introduce IST_SHSTK_SIZE (Name subject to improvement). * Skip writing the shadow stack token for !XEN_SHSTK builds. * Tweak clobbering to be correct and safe. --- xen/arch/x86/cpu/common.c | 24 ++++++++++++++++++++++++ xen/arch/x86/mm.c | 25 +++++++++++++++++++++++-- xen/include/asm-x86/config.h | 2 ++ xen/include/asm-x86/page.h | 1 + xen/include/asm-x86/processor.h | 3 ++- xen/include/asm-x86/x86_64/page.h | 1 + 6 files changed, 53 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index 690fd8baa8..dcc9ee08de 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -769,6 +769,30 @@ void load_system_tables(void) tss->rsp1 = 0x8600111111111111ul; tss->rsp2 = 0x8600111111111111ul; + /* Set up the shadow stack IST. */ + if (cpu_has_xen_shstk) { + volatile uint64_t *ist_ssp = this_cpu(tss_page).ist_ssp; + + /* + * Used entries must point at the supervisor stack token. + * Unused entries are poisoned. + * + * This IST Table may be live, and the NMI/#MC entries must + * remain valid on every instruction boundary, hence the + * volatile qualifier. + */ + ist_ssp[0] = 0x8600111111111111ul; + ist_ssp[IST_MCE] = stack_top + (IST_MCE * IST_SHSTK_SIZE) - 8; + ist_ssp[IST_NMI] = stack_top + (IST_NMI * IST_SHSTK_SIZE) - 8; + ist_ssp[IST_DB] = stack_top + (IST_DB * IST_SHSTK_SIZE) - 8; + ist_ssp[IST_DF] = stack_top + (IST_DF * IST_SHSTK_SIZE) - 8; + for ( i = IST_DF + 1; + i < ARRAY_SIZE(this_cpu(tss_page).ist_ssp); ++i ) + ist_ssp[i] = 0x8600111111111111ul; + + wrmsrl(MSR_INTERRUPT_SSP_TABLE, (unsigned long)ist_ssp); + } + BUILD_BUG_ON(sizeof(*tss) <= 0x67); /* Mandated by the architecture. */ _set_tssldt_desc(gdt + TSS_ENTRY, (unsigned long)tss, diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 2f1e716b6d..4d6d22cc41 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -5994,12 +5994,33 @@ void memguard_unguard_range(void *p, unsigned long l) #endif +static void write_sss_token(unsigned long *ptr) +{ + /* + * A supervisor shadow stack token is its own linear address, with the + * busy bit (0) clear. + */ + *ptr = (unsigned long)ptr; +} + void memguard_guard_stack(void *p) { - map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, _PAGE_NONE); + /* IST Shadow stacks. 4x 1k in stack page 0. */ + if ( IS_ENABLED(CONFIG_XEN_SHSTK) ) + { + write_sss_token(p + (IST_MCE * IST_SHSTK_SIZE) - 8); + write_sss_token(p + (IST_NMI * IST_SHSTK_SIZE) - 8); + write_sss_token(p + (IST_DB * IST_SHSTK_SIZE) - 8); + write_sss_token(p + (IST_DF * IST_SHSTK_SIZE) - 8); + } + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, PAGE_HYPERVISOR_SHSTK); + /* Primary Shadow Stack. 1x 4k in stack page 5. */ p += PRIMARY_SHSTK_SLOT * PAGE_SIZE; - map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, _PAGE_NONE); + if ( IS_ENABLED(CONFIG_XEN_SHSTK) ) + write_sss_token(p + PAGE_SIZE - 8); + + map_pages_to_xen((unsigned long)p, virt_to_mfn(p), 1, PAGE_HYPERVISOR_SHSTK); } void memguard_unguard_stack(void *p) diff --git a/xen/include/asm-x86/config.h b/xen/include/asm-x86/config.h index f3cf5df462..2ba234383d 100644 --- a/xen/include/asm-x86/config.h +++ b/xen/include/asm-x86/config.h @@ -66,6 +66,8 @@ #define STACK_ORDER 3 #define STACK_SIZE (PAGE_SIZE << STACK_ORDER) +#define IST_SHSTK_SIZE 1024 + #define TRAMPOLINE_STACK_SPACE PAGE_SIZE #define TRAMPOLINE_SPACE (KB(64) - TRAMPOLINE_STACK_SPACE) #define WAKEUP_STACK_MIN 3072 diff --git a/xen/include/asm-x86/page.h b/xen/include/asm-x86/page.h index 5acf3d3d5a..f632affaef 100644 --- a/xen/include/asm-x86/page.h +++ b/xen/include/asm-x86/page.h @@ -364,6 +364,7 @@ void efi_update_l4_pgtable(unsigned int l4idx, l4_pgentry_t); _PAGE_DIRTY | _PAGE_RW) #define __PAGE_HYPERVISOR_UCMINUS (__PAGE_HYPERVISOR | _PAGE_PCD) #define __PAGE_HYPERVISOR_UC (__PAGE_HYPERVISOR | _PAGE_PCD | _PAGE_PWT) +#define __PAGE_HYPERVISOR_SHSTK (__PAGE_HYPERVISOR_RO | _PAGE_DIRTY) #define MAP_SMALL_PAGES _PAGE_AVAIL0 /* don't use superpages mappings */ diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index 8ab09cf7ed..859bd9e2ec 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -435,7 +435,8 @@ struct __packed tss64 { uint16_t :16, bitmap; }; struct tss_page { - struct tss64 __aligned(PAGE_SIZE) tss; + uint64_t __aligned(PAGE_SIZE) ist_ssp[8]; + struct tss64 tss; }; DECLARE_PER_CPU(struct tss_page, tss_page); diff --git a/xen/include/asm-x86/x86_64/page.h b/xen/include/asm-x86/x86_64/page.h index 9876634881..26621f9519 100644 --- a/xen/include/asm-x86/x86_64/page.h +++ b/xen/include/asm-x86/x86_64/page.h @@ -171,6 +171,7 @@ static inline intpte_t put_pte_flags(unsigned int x) #define PAGE_HYPERVISOR_RW (__PAGE_HYPERVISOR_RW | _PAGE_GLOBAL) #define PAGE_HYPERVISOR_RX (__PAGE_HYPERVISOR_RX | _PAGE_GLOBAL) #define PAGE_HYPERVISOR_RWX (__PAGE_HYPERVISOR | _PAGE_GLOBAL) +#define PAGE_HYPERVISOR_SHSTK (__PAGE_HYPERVISOR_SHSTK | _PAGE_GLOBAL) #define PAGE_HYPERVISOR PAGE_HYPERVISOR_RW #define PAGE_HYPERVISOR_UCMINUS (__PAGE_HYPERVISOR_UCMINUS | \ From patchwork Wed May 27 19:18:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573805 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FAF0912 for ; Wed, 27 May 2020 19:19:52 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1613F2078C for ; Wed, 27 May 2020 19:19:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1613F2078C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aH-00042G-DG; Wed, 27 May 2020 19:19:25 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aF-00041n-QK for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:23 +0000 X-Inumbo-ID: f33f80e4-a04e-11ea-81bc-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f33f80e4-a04e-11ea-81bc-bc764e2007e4; Wed, 27 May 2020 19:19:13 +0000 (UTC) Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: Tq6+s3Zh7QaRK595E15ZFckIpIk5gndEZuDz8ttfd0qmFsoM1hkOEjbDpJmnafihXdS2HWfRcK QVkAVn3Rogew9UeKYWAH/IlD37FBiRZJR2anYYwrexQOk8+W6IdikgmgQ91IBXK3pEfLaP1DAM N6gd/Mt3pHXf98aiI/OYkIiMcPVmOwN4qrs210s1yoSsDlLfidrAwIRjL7qHSPnlzZBL0xbvuQ 67mO7Yp3pPLaQirrO8TluPLa7vsdyPssxO/H6VF+3sCQfFEywaUHox0NAOzp6g/8KgpVgvw+RP Q7A= X-SBRS: 2.7 X-MesageID: 18850558 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18850558" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 07/14] x86/cpu: Adjust enable_nmis() to be shadow stack compatible Date: Wed, 27 May 2020 20:18:40 +0100 Message-ID: <20200527191847.17207-8-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When executing an IRET-to-self, the shadow stack must agree with the regular stack. We can't manipulate SSP directly, so have to fake a shadow IRET frame by executing 3 CALLs, then editing the result to look correct. This is not a fastpath, is called on the BSP long before CET can be set up, and may be called on the crash path after CET is disabled. Use the fact that INCSSP is allocated from the hint nop space to construct a test for CET being active which is safe on all processors. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné --- xen/include/asm-x86/processor.h | 43 +++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/xen/include/asm-x86/processor.h b/xen/include/asm-x86/processor.h index 859bd9e2ec..badd7e60e5 100644 --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -545,17 +545,40 @@ static inline void enable_nmis(void) { unsigned long tmp; - asm volatile ( "mov %%rsp, %[tmp] \n\t" - "push %[ss] \n\t" - "push %[tmp] \n\t" - "pushf \n\t" - "push %[cs] \n\t" - "lea 1f(%%rip), %[tmp] \n\t" - "push %[tmp] \n\t" - "iretq; 1: \n\t" - : [tmp] "=&r" (tmp) + asm volatile ( "mov %%rsp, %[rsp] \n\t" + "lea .Ldone(%%rip), %[rip] \n\t" +#ifdef CONFIG_XEN_SHSTK + /* Check for CET-SS being active. */ + "mov $1, %k[ssp] \n\t" + "rdsspq %[ssp] \n\t" + "cmp $1, %k[ssp] \n\t" + "je .Lshstk_done \n\t" + + /* Push 3 words on the shadow stack */ + ".rept 3 \n\t" + "call 1f; nop; 1: \n\t" + ".endr \n\t" + + /* Fixup to be an IRET shadow stack frame */ + "wrssq %q[cs], -1*8(%[ssp]) \n\t" + "wrssq %[rip], -2*8(%[ssp]) \n\t" + "wrssq %[ssp], -3*8(%[ssp]) \n\t" + + ".Lshstk_done:" +#endif + /* Write an IRET regular frame */ + "push %[ss] \n\t" + "push %[rsp] \n\t" + "pushf \n\t" + "push %q[cs] \n\t" + "push %[rip] \n\t" + "iretq \n\t" + ".Ldone: \n\t" + : [rip] "=&r" (tmp), + [rsp] "=&r" (tmp), + [ssp] "=&r" (tmp) : [ss] "i" (__HYPERVISOR_DS), - [cs] "i" (__HYPERVISOR_CS) ); + [cs] "r" (__HYPERVISOR_CS) ); } void sysenter_entry(void); From patchwork Wed May 27 19:18:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573807 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B0D7912 for ; Wed, 27 May 2020 19:20:12 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 107AC207D3 for ; Wed, 27 May 2020 19:20:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 107AC207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1a7-0003zk-7p; Wed, 27 May 2020 19:19:15 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1a5-0003za-UD for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:13 +0000 X-Inumbo-ID: f25930ee-a04e-11ea-81bc-bc764e2007e4 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id f25930ee-a04e-11ea-81bc-bc764e2007e4; Wed, 27 May 2020 19:19:12 +0000 (UTC) Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: mbKeQ7QpZv3IAsquSsZ+NyeeuXXDsp5Ov4NaEfAcMJJ0LNNrOgQw0Ti7QwRh7vNC2an6Gox6Hs eZCEKK9hD4wSVxN90TtlQHL7WVq9ol/YPFFm6tWBpni7cPHg/oUYxNKCPBKRGjUCh2Ly6s3kvd JkNKKmQy5NMUDurQa7eM8aLs9WzGh9bZpl2MGsu0TCHcHBgh0ZFO9TZbxqmbVvDvPmMO3IqMK3 TmThn2U46kTojFfzBjApdDj6z4gcqzab3mVmIyHk0Nr5PHI8VhwC6cUGNhMq9Yk3lWBhiiMqqq RkQ= X-SBRS: 2.7 X-MesageID: 18850555 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18850555" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 08/14] x86/cpu: Adjust reset_stack_and_jump() to be shadow stack compatible Date: Wed, 27 May 2020 20:18:41 +0100 Message-ID: <20200527191847.17207-9-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" We need to unwind up to the supervisor token. See the comment for details. The use of UNLIKELY_END_SECTION in this case highlights that it isn't safe when it isn't the final statement of an asm(). Adjust all declarations with a newline. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Drop 'cmc' which was stray debugging. * Replace raw numbers with defines. * Use a real BUG frame in .fixup, to get static branch preduction working the right way around. --- xen/include/asm-x86/asm_defns.h | 8 +++---- xen/include/asm-x86/current.h | 48 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 49 insertions(+), 7 deletions(-) diff --git a/xen/include/asm-x86/asm_defns.h b/xen/include/asm-x86/asm_defns.h index b42a19b654..035708adac 100644 --- a/xen/include/asm-x86/asm_defns.h +++ b/xen/include/asm-x86/asm_defns.h @@ -177,13 +177,13 @@ register unsigned long current_stack_pointer asm("rsp"); #ifdef __clang__ /* clang's builtin assember can't do .subsection */ -#define UNLIKELY_START_SECTION ".pushsection .text.unlikely,\"ax\"" -#define UNLIKELY_END_SECTION ".popsection" +#define UNLIKELY_START_SECTION ".pushsection .text.unlikely,\"ax\"\n\t" +#define UNLIKELY_END_SECTION ".popsection\n\t" #else -#define UNLIKELY_START_SECTION ".subsection 1" -#define UNLIKELY_END_SECTION ".subsection 0" +#define UNLIKELY_START_SECTION ".subsection 1\n\t" +#define UNLIKELY_END_SECTION ".subsection 0\n\t" #endif diff --git a/xen/include/asm-x86/current.h b/xen/include/asm-x86/current.h index 99b66a0087..086326b81a 100644 --- a/xen/include/asm-x86/current.h +++ b/xen/include/asm-x86/current.h @@ -124,13 +124,55 @@ unsigned long get_stack_dump_bottom (unsigned long sp); # define CHECK_FOR_LIVEPATCH_WORK "" #endif +#ifdef CONFIG_XEN_SHSTK +/* + * We need to unwind the primary shadow stack to its supervisor token, located + * at 0x5ff8 from the base of the stack blocks. + * + * Read the shadow stack pointer, subtract it from 0x5ff8, divide by 8 to get + * the number of slots needing popping. + * + * INCSSPQ can't pop more than 255 entries. We shouldn't ever need to pop + * that many entries, and getting this wrong will cause us to #DF later. Turn + * it into a BUG() now for fractionally easier debugging. + */ +# define SHADOW_STACK_WORK \ + "mov $1, %[ssp];" \ + "rdsspd %[ssp];" \ + "cmp $1, %[ssp];" \ + "je .L_shstk_done.%=;" /* CET not active? Skip. */ \ + "mov $%c[skstk_base], %[val];" \ + "and $%c[stack_mask], %[ssp];" \ + "sub %[ssp], %[val];" \ + "shr $3, %[val];" \ + "cmp $255, %[val];" /* More than 255 entries? Crash. */ \ + UNLIKELY_START(a, shstk_adjust) \ + _ASM_BUGFRAME_TEXT(0) \ + UNLIKELY_END_SECTION \ + "incsspq %q[val];" \ + ".L_shstk_done.%=:" +#else +# define SHADOW_STACK_WORK "" +#endif + #define switch_stack_and_jump(fn, instr) \ ({ \ + unsigned int tmp; \ __asm__ __volatile__ ( \ - "mov %0,%%"__OP"sp;" \ + SHADOW_STACK_WORK \ + "mov %[stk], %%rsp;" \ instr \ - "jmp %c1" \ - : : "r" (guest_cpu_user_regs()), "i" (fn) : "memory" ); \ + "jmp %c[fun];" \ + : [val] "=&r" (tmp), \ + [ssp] "=&r" (tmp) \ + : [stk] "r" (guest_cpu_user_regs()), \ + [fun] "i" (fn), \ + [skstk_base] "i" \ + ((PRIMARY_SHSTK_SLOT + 1) * PAGE_SIZE - 8), \ + [stack_mask] "i" (STACK_SIZE - 1), \ + _ASM_BUGFRAME_INFO(BUGFRAME_bug, __LINE__, \ + __FILE__, NULL) \ + : "memory" ); \ unreachable(); \ }) From patchwork Wed May 27 19:18:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573815 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B0A7912 for ; Wed, 27 May 2020 19:20:20 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E51A7207D3 for ; Wed, 27 May 2020 19:20:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E51A7207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aH-00042T-MQ; Wed, 27 May 2020 19:19:25 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1aG-00041w-2S for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:19:24 +0000 X-Inumbo-ID: f518fc1a-a04e-11ea-a777-12813bfff9fa Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f518fc1a-a04e-11ea-a777-12813bfff9fa; Wed, 27 May 2020 19:19:17 +0000 (UTC) Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: AqhgHtvucTXh9gKrfysPQjfVwMB0aAHR29VpIB06eomiAmpiSuq6TuZPrORn6qj4+x6hCO6gd+ SVfFlrKExbY8F3Atho7vCyhxi4fAZnu21BA2HyMi3ltCqkKkS4O8jkHQ0LfIaBT+ViAzXYjlRf Xck5nRfYrDK76ervV5In3SK6anRX5eygGGYsEbTEJFpVcojs6VfxwOnIpP2Znpfl3pX2t9h21I tR35FF9r5Bug6GxwSAg4sHJIuukG+UQV578hX0usflysaIbNriaUFgS9dDOVaDrsVJHsfWh3+3 0vA= X-SBRS: 2.7 X-MesageID: 19333922 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="19333922" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 09/14] x86/spec-ctrl: Adjust DO_OVERWRITE_RSB to be shadow stack compatible Date: Wed, 27 May 2020 20:18:42 +0100 Message-ID: <20200527191847.17207-10-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The 32 calls need dropping from the shadow stack as well as the regular stack. To shorten the code, we can use the 32bit forms of RDSSP/INCSSP, but need to double up the input to INCSSP to counter the operand size based multiplier. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné --- xen/include/asm-x86/spec_ctrl_asm.h | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/xen/include/asm-x86/spec_ctrl_asm.h b/xen/include/asm-x86/spec_ctrl_asm.h index c60093b090..cb34299a86 100644 --- a/xen/include/asm-x86/spec_ctrl_asm.h +++ b/xen/include/asm-x86/spec_ctrl_asm.h @@ -83,9 +83,9 @@ * Requires nothing * Clobbers \tmp (%rax by default), %rcx * - * Requires 256 bytes of stack space, but %rsp has no net change. Based on - * Google's performance numbers, the loop is unrolled to 16 iterations and two - * calls per iteration. + * Requires 256 bytes of {,shadow}stack space, but %rsp/SSP has no net + * change. Based on Google's performance numbers, the loop is unrolled to 16 + * iterations and two calls per iteration. * * The call filling the RSB needs a nonzero displacement. A nop would do, but * we use "1: pause; lfence; jmp 1b" to safely contains any ret-based @@ -114,6 +114,16 @@ sub $1, %ecx jnz .L\@_fill_rsb_loop mov %\tmp, %rsp /* Restore old %rsp */ + +#ifdef CONFIG_XEN_SHSTK + mov $1, %ecx + rdsspd %ecx + cmp $1, %ecx + je .L\@_shstk_done + mov $64, %ecx /* 64 * 4 bytes, given incsspd */ + incsspd %ecx /* Restore old SSP */ +.L\@_shstk_done: +#endif .endm .macro DO_SPEC_CTRL_ENTRY_FROM_HVM From patchwork Wed May 27 19:18:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D2848912 for ; Wed, 27 May 2020 19:35:30 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B9492207D3 for ; Wed, 27 May 2020 19:35:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B9492207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1oy-0006cq-2D; Wed, 27 May 2020 19:34:36 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1ow-0006cN-HW for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:34:34 +0000 X-Inumbo-ID: 130007e4-a051-11ea-81bc-bc764e2007e4 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 130007e4-a051-11ea-81bc-bc764e2007e4; Wed, 27 May 2020 19:34:26 +0000 (UTC) Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: NPahr4HOwB/rPxUenW5oMXduKyL+uOiKhCpIDIFNdBdKGst4pFLbAcGaUzrmUK8S5AUttl92jn apZJLDNuAGxnq7nvb+RwyoDq0h6KJ9aNV5JKxkAqDGy05fCpCQYS+sW5Infn+BhObaVgccfRD7 Cniq80QEBLLaA/PichBP8Kol8/FhBO7RzKG9oVCKq0naYVLJyYLSV1aBHMrJLVBP/cEpRLYDvF Gyf0yoFxFWl5qT/uT969foSfPYFHrdrSEZ43ePELaNMyryXcW9PQ+rV9YGfN093+Y80M8uaY2f f+w= X-SBRS: 2.7 X-MesageID: 18592446 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18592446" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 10/14] x86/extable: Adjust extable handling to be shadow stack compatible Date: Wed, 27 May 2020 20:18:43 +0100 Message-ID: <20200527191847.17207-11-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When adjusting an IRET frame to recover from a fault, and equivalent adjustment needs making in the shadow IRET frame. The adjustment in exception_with_ints_disabled() could in principle be an alternative block rather than an ifdef, as the only two current users of _PRE_EXTABLE() are IRET-to-guest instructions. However, this is not a fastpath, and this form is more robust to future changes. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Break extable_shstk_fixup() out into a separate function. * Guard from shstk underflows, and unrealistic call traces. --- xen/arch/x86/traps.c | 67 ++++++++++++++++++++++++++++++++++++++++++++- xen/arch/x86/x86_64/entry.S | 11 +++++++- 2 files changed, 76 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index 235a72cf4a..ce910294ea 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -363,7 +363,7 @@ static void show_guest_stack(struct vcpu *v, const struct cpu_user_regs *regs) } /* - * Notes for get_stack_trace_bottom() and get_stack_dump_bottom() + * Notes for get_{stack,shstk}*_bottom() helpers * * Stack pages 1 - 4: * These are all 1-page IST stacks. Each of these stacks have an exception @@ -400,6 +400,18 @@ unsigned long get_stack_trace_bottom(unsigned long sp) } } +static unsigned long get_shstk_bottom(unsigned long sp) +{ + switch ( get_stack_page(sp) ) + { +#ifdef CONFIG_XEN_SHSTK + case 0: return ROUNDUP(sp, IST_SHSTK_SIZE) - sizeof(unsigned long); + case 5: return ROUNDUP(sp, PAGE_SIZE) - sizeof(unsigned long); +#endif + default: return sp - sizeof(unsigned long); + } +} + unsigned long get_stack_dump_bottom(unsigned long sp) { switch ( get_stack_page(sp) ) @@ -763,6 +775,56 @@ static void do_reserved_trap(struct cpu_user_regs *regs) trapnr, vec_name(trapnr), regs->error_code); } +static void extable_shstk_fixup(struct cpu_user_regs *regs, unsigned long fixup) +{ + unsigned long ssp, *ptr, *base; + + asm ( "rdsspq %0" : "=r" (ssp) : "0" (1) ); + if ( ssp == 1 ) + return; + + ptr = _p(ssp); + base = _p(get_shstk_bottom(ssp)); + + for ( ; ptr < base; ++ptr ) + { + /* + * Search for %rip. The shstk currently looks like this: + * + * ... [Likely pointed to by SSP] + * %cs [== regs->cs] + * %rip [== regs->rip] + * SSP [Likely points to 3 slots higher, above %cs] + * ... [call tree to this function, likely 2/3 slots] + * + * and we want to overwrite %rip with fixup. There are two + * complications: + * 1) We cant depend on SSP values, because they won't differ by 3 + * slots if the exception is taken on an IST stack. + * 2) There are synthetic (unrealistic but not impossible) scenarios + * where %rip can end up in the call tree to this function, so we + * can't check against regs->rip alone. + * + * Check for both reg->rip and regs->cs matching. + */ + + if ( ptr[0] == regs->rip && ptr[1] == regs->cs ) + { + asm ( "wrssq %[fix], %[stk]" + : [stk] "=m" (*ptr) + : [fix] "r" (fixup) ); + return; + } + } + + /* + * We failed to locate and fix up the shadow IRET frame. This could be + * due to shadow stack corruption, or bad logic above. We cannot continue + * executing the interrupted context. + */ + BUG(); +} + static bool extable_fixup(struct cpu_user_regs *regs, bool print) { unsigned long fixup = search_exception_table(regs); @@ -779,6 +841,9 @@ static bool extable_fixup(struct cpu_user_regs *regs, bool print) vec_name(regs->entry_vector), regs->error_code, _p(regs->rip), _p(regs->rip), _p(fixup)); + if ( IS_ENABLED(CONFIG_XEN_SHSTK) ) + extable_shstk_fixup(regs, fixup); + regs->rip = fixup; this_cpu(last_extable_addr) = regs->rip; diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S index f7ee3dce91..78ac0df49f 100644 --- a/xen/arch/x86/x86_64/entry.S +++ b/xen/arch/x86/x86_64/entry.S @@ -708,7 +708,16 @@ exception_with_ints_disabled: call search_pre_exception_table testq %rax,%rax # no fixup code for faulting EIP? jz 1b - movq %rax,UREGS_rip(%rsp) + movq %rax,UREGS_rip(%rsp) # fixup regular stack + +#ifdef CONFIG_XEN_SHSTK + mov $1, %edi + rdsspq %rdi + cmp $1, %edi + je .L_exn_shstk_done + wrssq %rax, (%rdi) # fixup shadow stack +.L_exn_shstk_done: +#endif subq $8,UREGS_rsp(%rsp) # add ec/ev to previous stack frame testb $15,UREGS_rsp(%rsp) # return %rsp is now aligned? jz 1f # then there is a pad quadword already From patchwork Wed May 27 19:18:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573829 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69702912 for ; Wed, 27 May 2020 19:34:56 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4FB5F207D3 for ; Wed, 27 May 2020 19:34:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4FB5F207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1on-0006a7-Hc; Wed, 27 May 2020 19:34:25 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1om-0006Zn-Lz for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:34:24 +0000 X-Inumbo-ID: 11666ca2-a051-11ea-a779-12813bfff9fa Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 11666ca2-a051-11ea-a779-12813bfff9fa; Wed, 27 May 2020 19:34:23 +0000 (UTC) Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: wY10bV6tZiw8sRMAEjklee9H8RWGt6pqPgLjQ3cvNmM25+VfjtEE08bM7dz9/CBxYagtPn4LQ9 9ZPonaohusmVZGlVGlaSY1fLc4vEQzXgXV3Yr6ZDI1uZZZPDRKuR+aRqwlEOWGnSBSOOHK7054 CkoiN1AupubV16bKGGtSqGLVA6RuL/rHFOrdGnXEKATOiFpGSoHyhCb4cNaAUOdEKdru+jFRi6 EEuEuMhCcHhljANqiK7tKQo2J3/9jRGidOEUY09coR0Ek2+tzdSE5HXOIj+lJGwhLT0gjFgi0I pzw= X-SBRS: 2.7 X-MesageID: 18615173 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18615173" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 11/14] x86/alt: Adjust _alternative_instructions() to not create shadow stacks Date: Wed, 27 May 2020 20:18:44 +0100 Message-ID: <20200527191847.17207-12-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The current alternatives algorithm clears CR0.WP and writes into .text. This has a side effect of the mappings becoming shadow stacks once CET is active. Adjust _alternative_instructions() to clean up after itself. This involves extending the set of bits modify_xen_mappings() to include Dirty (and Accessed for good measure). Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné --- xen/arch/x86/alternative.c | 14 ++++++++++++++ xen/arch/x86/mm.c | 6 +++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/alternative.c b/xen/arch/x86/alternative.c index ce2b4302e6..004e9ede25 100644 --- a/xen/arch/x86/alternative.c +++ b/xen/arch/x86/alternative.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -398,6 +399,19 @@ static void __init _alternative_instructions(bool force) panic("Timed out waiting for alternatives self-NMI to hit\n"); set_nmi_callback(saved_nmi_callback); + + /* + * When Xen is using shadow stacks, the alternatives clearing CR0.WP and + * writing into the mappings set dirty bits, turning the mappings into + * shadow stack mappings. + * + * While we can execute from them, this would also permit them to be the + * target of WRSS instructions, so reset the dirty after patching. + */ + if ( cpu_has_xen_shstk ) + modify_xen_mappings(XEN_VIRT_START + MB(2), + (unsigned long)&__2M_text_end, + PAGE_HYPERVISOR_RX); } void __init alternative_instructions(void) diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 4d6d22cc41..711b12828f 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -5442,8 +5442,8 @@ int populate_pt_range(unsigned long virt, unsigned long nr_mfns) * mappings, but will shatter superpages if necessary, and will destroy * mappings if not passed _PAGE_PRESENT. * - * The only flags considered are NX, RW and PRESENT. All other input flags - * are ignored. + * The only flags considered are NX, D, A, RW and PRESENT. All other input + * flags are ignored. * * It is an error to call with present flags over an unpopulated range. */ @@ -5456,7 +5456,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, unsigned int nf) unsigned long v = s; /* Set of valid PTE bits which may be altered. */ -#define FLAGS_MASK (_PAGE_NX|_PAGE_RW|_PAGE_PRESENT) +#define FLAGS_MASK (_PAGE_NX|_PAGE_DIRTY|_PAGE_ACCESSED|_PAGE_RW|_PAGE_PRESENT) nf &= FLAGS_MASK; ASSERT(IS_ALIGNED(s, PAGE_SIZE)); From patchwork Wed May 27 19:18:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573835 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1BED14C0 for ; Wed, 27 May 2020 19:35:36 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 984522078B for ; Wed, 27 May 2020 19:35:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 984522078B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1oj-0006Ze-1G; Wed, 27 May 2020 19:34:21 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1oh-0006ZZ-NH for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:34:19 +0000 X-Inumbo-ID: 0e5e31e8-a051-11ea-8993-bc764e2007e4 Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 0e5e31e8-a051-11ea-8993-bc764e2007e4; Wed, 27 May 2020 19:34:18 +0000 (UTC) Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 1Fy0vTmIqdblFkHjLrUVZU3/BmKXzCKiM8pMK5MITBvTnvTgIv0MZhoxUPvTF3MWxpbVwfe6yi 1gGDaBuP2Bi6nW0jfWOzXb0YE1UD+QTOupodLlegaGD1PVAf7X3YHY1n88wHDOxoAtmerMJCPm iEJxzbXvy/KcjtqHr3kOOkspOJTy/m88xlA3kywA/KvGrY3mKVmqihgZcNDjCu9LZ35jc/0Pea XqNB90JuXdcgrbb2mjNujuLyhUuc60vtdpYi36yQPh1BhtCVt19cTDBfnIZryDjbGRe5mqJbfi Ezc= X-SBRS: 2.7 X-MesageID: 18947884 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18947884" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 12/14] x86/entry: Adjust guest paths to be shadow stack compatible Date: Wed, 27 May 2020 20:18:45 +0100 Message-ID: <20200527191847.17207-13-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The SYSCALL/SYSENTER/SYSRET paths need to use {SET,CLR}SSBSY. The IRET to guest paths must not. In the SYSRET path, re-position the mov which loads rip into %rcx so we can use %rcx for CLRSSBSY, rather than spilling another register to the stack. While we can in principle detect shadow stack corruption and a failure to clear the supervisor token busy bit in the SYSRET path (by inspecting the carry flag following CLRSSBSY), we cannot detect similar problems for the IRET path (IRET is specified not to fault in this case). We will double fault at some point later, when next trying to enter Xen, due to an already-set supervisor shadow stack busy bit. As SYSRET is a uncommon path anyway, avoid the added complexity for no appreciable gain. The IST switch onto the primary stack is not great as we have an instruction boundary with no shadow stack. This is the least bad option available. These paths are not used before shadow stacks are properly established, so can use alternatives to avoid extra runtime CET detection logic. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Break comment deletion out to an earlier patch * SETSSBSY on the SYSENTER path as well * Don't spill %rax to the stack in the SYSRET path --- xen/arch/x86/x86_64/compat/entry.S | 1 + xen/arch/x86/x86_64/entry.S | 32 +++++++++++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/x86_64/compat/entry.S b/xen/arch/x86/x86_64/compat/entry.S index 3cd375bd48..2ca81341a4 100644 --- a/xen/arch/x86/x86_64/compat/entry.S +++ b/xen/arch/x86/x86_64/compat/entry.S @@ -198,6 +198,7 @@ ENTRY(cr4_pv32_restore) /* See lstar_enter for entry register state. */ ENTRY(cstar_enter) + ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK /* sti could live here when we don't switch page tables below. */ CR4_PV32_RESTORE movq 8(%rsp),%rax /* Restore %rax. */ diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S index 78ac0df49f..449ee468e4 100644 --- a/xen/arch/x86/x86_64/entry.S +++ b/xen/arch/x86/x86_64/entry.S @@ -191,9 +191,16 @@ restore_all_guest: sarq $47,%rcx incl %ecx cmpl $1,%ecx - movq 8(%rsp),%rcx # RIP ja iret_exit_to_guest + /* Clear the supervisor shadow stack token busy bit. */ +.macro rag_clrssbsy + rdsspq %rcx + clrssbsy (%rcx) +.endm + ALTERNATIVE "", rag_clrssbsy, X86_FEATURE_XEN_SHSTK + + movq 8(%rsp), %rcx # RIP cmpw $FLAT_USER_CS32,16(%rsp)# CS movq 32(%rsp),%rsp # RSP je 1f @@ -226,6 +233,7 @@ iret_exit_to_guest: * %ss must be saved into the space left by the trampoline. */ ENTRY(lstar_enter) + ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK /* sti could live here when we don't switch page tables below. */ movq 8(%rsp),%rax /* Restore %rax. */ movq $FLAT_KERNEL_SS,8(%rsp) @@ -259,6 +267,7 @@ ENTRY(lstar_enter) jmp test_all_events ENTRY(sysenter_entry) + ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK /* sti could live here when we don't switch page tables below. */ pushq $FLAT_USER_SS pushq $0 @@ -877,6 +886,27 @@ handle_ist_exception: movl $UREGS_kernel_sizeof/8,%ecx movq %rdi,%rsp rep movsq + + /* Switch Shadow Stacks */ +.macro ist_switch_shstk + rdsspq %rdi + clrssbsy (%rdi) + /* + * Switching supervisor shadow stacks is specially hard, as supervisor + * and restore tokens are incompatible. + * + * For now, we only need to switch on to an unused primary shadow + * stack, so use SETSSBSY for the purpose, exactly like the + * SYSCALL/SYSENTER entry. + * + * Ideally, we'd want to CLRSSBSY after switching stacks, but that + * will leave SSP zeroed so it not an option. Instead, we transiently + * have a zero SSP on this instruction boundary, and depend on IST for + * NMI/#MC protection. + */ + setssbsy +.endm + ALTERNATIVE "", ist_switch_shstk, X86_FEATURE_XEN_SHSTK 1: #else ASSERT_CONTEXT_IS_XEN From patchwork Wed May 27 19:18:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80B5514B7 for ; Wed, 27 May 2020 19:35:24 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 66B8D207D3 for ; Wed, 27 May 2020 19:35:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 66B8D207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1os-0006bS-Q3; Wed, 27 May 2020 19:34:30 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1or-0006b7-HC for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:34:29 +0000 X-Inumbo-ID: 107ad8aa-a051-11ea-9947-bc764e2007e4 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 107ad8aa-a051-11ea-9947-bc764e2007e4; Wed, 27 May 2020 19:34:21 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: y3yuGKwlIoZ/HdGOnxeDgPPOs0wYll11M5AmTmCsBOgUvsK+Ui34zh3c7zEMnp8C8ZMwhgrbzY ONgNIr9ifCCY9QmRQ592wMWvYsy952D5rLRirgbTZYlYR7XasUsPjGm0wkdLgkfMMZz9RoCyEk inT5OCYK5wlOdtiHr/frvxX95qgNFOpt1MgEG3BNf+hFvXhmr9lkj8kxbiXv/7g5I5pd+4W6eV wrqjo/oLEFZOBWyrfmG/mmdmOIKhu9oh7cWsx4x6hWghUHzJNzOn9MiviLGQ+LzQgofCXVJXL3 6Vg= X-SBRS: 2.7 X-MesageID: 18890134 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18890134" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 13/14] x86/S3: Save and restore Shadow Stack configuration Date: Wed, 27 May 2020 20:18:46 +0100 Message-ID: <20200527191847.17207-14-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" See code for details Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné Semi-RFC - I can't actually test this path. Currently attempting to arrange for someone else to. v2: * New, split out of "x86/shstk: Activate Supervisor Shadow Stacks" * Drop asm/config.h include * Fix order of operations to avoid multiple crashes. --- xen/arch/x86/acpi/wakeup_prot.S | 58 +++++++++++++++++++++++++++++++++++++++++ xen/include/asm-x86/msr-index.h | 3 +++ xen/include/asm-x86/x86-defns.h | 1 + 3 files changed, 62 insertions(+) diff --git a/xen/arch/x86/acpi/wakeup_prot.S b/xen/arch/x86/acpi/wakeup_prot.S index 4dba6020a7..dcc7e2327d 100644 --- a/xen/arch/x86/acpi/wakeup_prot.S +++ b/xen/arch/x86/acpi/wakeup_prot.S @@ -1,3 +1,7 @@ +#include +#include +#include + .file __FILE__ .text .code64 @@ -15,6 +19,12 @@ ENTRY(do_suspend_lowlevel) mov %cr0, %rax mov %rax, saved_cr0(%rip) +#ifdef CONFIG_XEN_SHSTK + mov $1, %eax + rdsspq %rax + mov %rax, saved_ssp(%rip) +#endif + /* enter sleep state physically */ mov $3, %edi call acpi_enter_sleep_state @@ -48,6 +58,51 @@ ENTRY(s3_resume) pushq %rax lretq 1: +#ifdef CONFIG_XEN_SHSTK + /* + * Restoring SSP is a little complicated, because we are intercepting + * an in-use shadow stack. Write a temporary token under the stack, + * so SETSSBSY will successfully load a value useful for us, then + * reset MSR_PL0_SSP to its usual value and pop the temporary token. + */ + mov saved_rsp(%rip), %rdi + cmpq $1, %rdi + je .L_shstk_done + + /* Set up MSR_S_CET. */ + mov $MSR_S_CET, %ecx + xor %edx, %edx + mov $CET_SHSTK_EN | CET_WRSS_EN, %eax + wrmsr + + /* Construct the temporary supervisor token under SSP. */ + sub $8, %rdi + + /* Load it into MSR_PL0_SSP. */ + mov $MSR_PL0_SSP, %ecx + mov %rdi, %rdx + shr $32, %rdx + mov %edi, %eax + wrmsr + + /* Enable CET. MSR_INTERRUPT_SSP_TABLE is set up later in load_system_tables(). */ + mov $XEN_MINIMAL_CR4 | X86_CR4_CET, %ebx + mov %rbx, %cr4 + + /* Write the temporary token onto the shadow stack, and activate it. */ + wrssq %rdi, (%rdi) + setssbsy + + /* Reset MSR_PL0_SSP back to its normal value. */ + and $~(STACK_SIZE - 1), %eax + or $(PRIMARY_SHSTK_SLOT + 1) * PAGE_SIZE - 8, %eax + wrmsr + + /* Pop the temporary token off the stack. */ + mov $2, %eax + incsspd %eax +.L_shstk_done: +#endif call load_system_tables @@ -65,6 +120,9 @@ ENTRY(s3_resume) saved_rsp: .quad 0 saved_cr0: .quad 0 +#ifdef CONFIG_XEN_SHSTK +saved_ssp: .quad 0 +#endif GLOBAL(saved_magic) .long 0x9abcdef0 diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-index.h index 85c5f20b76..cdfb7b047b 100644 --- a/xen/include/asm-x86/msr-index.h +++ b/xen/include/asm-x86/msr-index.h @@ -68,6 +68,9 @@ #define MSR_U_CET 0x000006a0 #define MSR_S_CET 0x000006a2 +#define CET_SHSTK_EN (_AC(1, ULL) << 0) +#define CET_WRSS_EN (_AC(1, ULL) << 1) + #define MSR_PL0_SSP 0x000006a4 #define MSR_PL1_SSP 0x000006a5 #define MSR_PL2_SSP 0x000006a6 diff --git a/xen/include/asm-x86/x86-defns.h b/xen/include/asm-x86/x86-defns.h index 5366e2d018..072c87042c 100644 --- a/xen/include/asm-x86/x86-defns.h +++ b/xen/include/asm-x86/x86-defns.h @@ -73,6 +73,7 @@ #define X86_CR4_SMEP 0x00100000 /* enable SMEP */ #define X86_CR4_SMAP 0x00200000 /* enable SMAP */ #define X86_CR4_PKE 0x00400000 /* enable PKE */ +#define X86_CR4_CET 0x00800000 /* Control-flow Enforcement Technology */ /* * XSTATE component flags in XCR0 From patchwork Wed May 27 19:18:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11573839 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29F7214B7 for ; Wed, 27 May 2020 19:35:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1067B2078B for ; Wed, 27 May 2020 19:35:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1067B2078B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1on-0006Zx-9F; Wed, 27 May 2020 19:34:25 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1je1om-0006Zk-GC for xen-devel@lists.xenproject.org; Wed, 27 May 2020 19:34:24 +0000 X-Inumbo-ID: 0faab7f6-a051-11ea-9947-bc764e2007e4 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 0faab7f6-a051-11ea-9947-bc764e2007e4; Wed, 27 May 2020 19:34:20 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 63PRrOCGIORe2oeMNymVJruyxbUTOeT4sjMpQqtroN19UX8P7qCkVtoTz54iDIaAiqLt8qi9Mt /mY5IUfUBqYTY0xurY8LU5jdhtOp6HuTYlEiklfG2DC9LhZY0K3tUy6xmWj8ODIDd1+1pljwbE J0alG1ldPodeuqZw1ayxiyajiThrKxl/Yfsk9Yvwobj9x3FG8Oag6vtyxB+8BDQ/rV4DJ0Mpy3 ktkAj6jGI3H2I4XnI6XxUP8E6NFqbejvopVIRC3Kd3z7D47rXDJtqwsnXGYxIGD+f/2mpjbITz QWA= X-SBRS: 2.7 X-MesageID: 18890129 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.73,442,1583211600"; d="scan'208";a="18890129" From: Andrew Cooper To: Xen-devel Subject: [PATCH v2 14/14] x86/shstk: Activate Supervisor Shadow Stacks Date: Wed, 27 May 2020 20:18:47 +0100 Message-ID: <20200527191847.17207-15-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200527191847.17207-1-andrew.cooper3@citrix.com> References: <20200527191847.17207-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" With all other plumbing in place, activate shadow stacks when possible. Note that this requires prohibiting the use of PV32. Compatibility can be maintained if necessary via PV-Shim. The BSP needs to wait until alternatives have run (to avoid interaction with CR0.WP), and after the first reset_stack_and_jump() to avoid having a pristine shadow stack interact in problematic ways with an in-use regular stack. Activate shadow stack in reinit_bsp_stack(). APs have all infrastructure set up by the booting CPU, so enable shadow stacks before entering C. Adjust the logic to call start_secondary rather than jump to it, so stack traces make more sense. The crash path needs to turn CET off to avoid interfering with the crash kernel's environment. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné v2: * Split S3 out into earlier patch. * Discuss CET-SS disabling PV32, and start_secondary to be a call. --- docs/misc/xen-command-line.pandoc | 8 ++++++++ xen/arch/x86/boot/x86_64.S | 35 +++++++++++++++++++++++++++++++++-- xen/arch/x86/cpu/common.c | 5 +++++ xen/arch/x86/crash.c | 7 +++++++ xen/arch/x86/setup.c | 26 ++++++++++++++++++++++++++ xen/arch/x86/spec_ctrl.c | 8 ++++++++ 6 files changed, 87 insertions(+), 2 deletions(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index d4934eabb7..9892c57ee9 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -287,6 +287,10 @@ call/jmp COP/JOP) attacks. `cet=no-shstk` will cause Xen not to use Shadow Stacks even when support is available in hardware. + Shadow Stacks are incompatible with 32bit PV guests. This option will + override the `pv=32` boolean to false. Backwards compatibility can be + maintained with the `pv-shim` mechanism. + ### clocksource (x86) > `= pit | hpet | acpi | tsc` @@ -1726,6 +1730,10 @@ Controls for aspects of PV guest support. * The `32` boolean controls whether 32bit PV guests can be created. It defaults to `true`, and is ignored when `CONFIG_PV32` is compiled out. + 32bit PV guests are incompatible with CET Shadow Stacks. If Xen is using + shadow stacks, this option will be overridden to `false`. Backwards + compatibility can be maintained with the `pv-shim` mechanism. + ### pv-linear-pt (x86) > `= ` diff --git a/xen/arch/x86/boot/x86_64.S b/xen/arch/x86/boot/x86_64.S index 314a32a19f..551acd9e94 100644 --- a/xen/arch/x86/boot/x86_64.S +++ b/xen/arch/x86/boot/x86_64.S @@ -28,8 +28,39 @@ ENTRY(__high_start) lretq 1: test %ebx,%ebx - jnz start_secondary - + jz .L_bsp + + /* APs. Set up shadow stacks before entering C. */ + + testl $cpufeat_mask(X86_FEATURE_XEN_SHSTK), \ + CPUINFO_FEATURE_OFFSET(X86_FEATURE_XEN_SHSTK) + boot_cpu_data(%rip) + je .L_ap_shstk_done + + /* Set up MSR_S_CET. */ + mov $MSR_S_CET, %ecx + xor %edx, %edx + mov $CET_SHSTK_EN | CET_WRSS_EN, %eax + wrmsr + + /* Derive MSR_PL0_SSP from %rsp (token written when stack is allocated). */ + mov $MSR_PL0_SSP, %ecx + mov %rsp, %rdx + shr $32, %rdx + mov %esp, %eax + and $~(STACK_SIZE - 1), %eax + or $(PRIMARY_SHSTK_SLOT + 1) * PAGE_SIZE - 8, %eax + wrmsr + + /* Enable CET. MSR_INTERRUPT_SSP_TABLE is set up later in load_system_tables(). */ + mov $XEN_MINIMAL_CR4 | X86_CR4_CET, %ecx + mov %rcx, %cr4 + setssbsy + +.L_ap_shstk_done: + call start_secondary + BUG /* start_secondary() shouldn't return. */ + +.L_bsp: /* Pass off the Multiboot info structure to C land (if applicable). */ mov multiboot_ptr(%rip),%edi call __start_xen diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index dcc9ee08de..b4416f941c 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -329,6 +329,11 @@ void __init early_cpu_init(void) x86_cpuid_vendor_to_str(c->x86_vendor), c->x86, c->x86, c->x86_model, c->x86_model, c->x86_mask, eax); + if (c->cpuid_level >= 7) { + cpuid_count(7, 0, &eax, &ebx, &ecx, &edx); + c->x86_capability[cpufeat_word(X86_FEATURE_CET_SS)] = ecx; + } + eax = cpuid_eax(0x80000000); if ((eax >> 16) == 0x8000 && eax >= 0x80000008) { eax = cpuid_eax(0x80000008); diff --git a/xen/arch/x86/crash.c b/xen/arch/x86/crash.c index 450eecd46b..0611b4fb9b 100644 --- a/xen/arch/x86/crash.c +++ b/xen/arch/x86/crash.c @@ -200,6 +200,13 @@ void machine_crash_shutdown(void) /* Reset CPUID masking and faulting to the host's default. */ ctxt_switch_levelling(NULL); + /* Disable shadow stacks. */ + if ( cpu_has_xen_shstk ) + { + wrmsrl(MSR_S_CET, 0); + write_cr4(read_cr4() & ~X86_CR4_CET); + } + info = kexec_crash_save_info(); info->xen_phys_start = xen_phys_start; info->dom0_pfn_to_mfn_frame_list_list = diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 584589baff..0c4fd8c6a8 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c @@ -664,6 +664,13 @@ static void __init noreturn reinit_bsp_stack(void) stack_base[0] = stack; memguard_guard_stack(stack); + if ( cpu_has_xen_shstk ) + { + wrmsrl(MSR_PL0_SSP, (unsigned long)stack + 0x5ff8); + wrmsrl(MSR_S_CET, CET_SHSTK_EN | CET_WRSS_EN); + asm volatile ("setssbsy" ::: "memory"); + } + reset_stack_and_jump_nolp(init_done); } @@ -1065,6 +1072,21 @@ void __init noreturn __start_xen(unsigned long mbi_p) /* This must come before e820 code because it sets paddr_bits. */ early_cpu_init(); + /* Choose shadow stack early, to set infrastructure up appropriately. */ + if ( opt_xen_shstk && boot_cpu_has(X86_FEATURE_CET_SS) ) + { + printk("Enabling Supervisor Shadow Stacks\n"); + + setup_force_cpu_cap(X86_FEATURE_XEN_SHSTK); +#ifdef CONFIG_PV32 + if ( opt_pv32 ) + { + opt_pv32 = 0; + printk(" - Disabling PV32 due to Shadow Stacks\n"); + } +#endif + } + /* Sanitise the raw E820 map to produce a final clean version. */ max_page = raw_max_page = init_e820(memmap_type, &e820_raw); @@ -1801,6 +1823,10 @@ void __init noreturn __start_xen(unsigned long mbi_p) alternative_branches(); + /* Defer CR4.CET until alternatives have finished playing with CR4.WP */ + if ( cpu_has_xen_shstk ) + set_in_cr4(X86_CR4_CET); + /* * NB: when running as a PV shim VCPUOP_up/down is wired to the shim * physical cpu_add/remove functions, so launch the guest with only diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c index c5d8e587a8..a94be2d594 100644 --- a/xen/arch/x86/spec_ctrl.c +++ b/xen/arch/x86/spec_ctrl.c @@ -882,6 +882,14 @@ void __init init_speculation_mitigations(void) hw_smt_enabled = check_smt_enabled(); /* + * First, disable the use of retpolines if Xen is using shadow stacks, as + * they are incompatible. + */ + if ( cpu_has_xen_shstk && + (opt_thunk == THUNK_DEFAULT || opt_thunk == THUNK_RETPOLINE) ) + thunk = THUNK_JMP; + + /* * Has the user specified any custom BTI mitigations? If so, follow their * instructions exactly and disable all heuristics. */