From patchwork Tue Jun 14 14:33:12 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 9175961 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7C8DB60772 for ; Tue, 14 Jun 2016 14:35:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D1262012F for ; Tue, 14 Jun 2016 14:35:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61D1F2824A; Tue, 14 Jun 2016 14:35:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CA9EC2012F for ; Tue, 14 Jun 2016 14:35:27 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bCpP6-00037k-1O; Tue, 14 Jun 2016 14:33:20 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bCpP5-00037a-5Y for xen-devel@lists.xenproject.org; Tue, 14 Jun 2016 14:33:19 +0000 Received: from [85.158.139.211] by server-4.bemta-5.messagelabs.com id 0B/62-11823-EA510675; Tue, 14 Jun 2016 14:33:18 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAIsWRWlGSWpSXmKPExsXS6fjDS3etaEK 4wdxXmhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8aE1e+ZCvqtKh4fOMDawNis38XIySEkkCfx 6+EXVhCbV8BO4mnvTHYQW0LAUGLf/FVsIDaLgKrEtyn9YHE2AXWJtmfbgeo5OEQEDCTOHU0CM ZkF9CW2rWMBqRAWsJLYdfIwC0iYV0BQ4u8OYZAwM9DwDZv6GScwcs1CyMxCkoGwtSQe/rrFAm FrSyxb+Jp5Fth8aYnl/zggwg4SJ69tQ1MCYntL3Npxk2kBI8cqRo3i1KKy1CJdQyO9pKLM9Iy S3MTMHF1DA1O93NTi4sT01JzEpGK95PzcTYzA0GMAgh2MfbOcDzFKcjApifJuZU4IF+JLyk+p zEgszogvKs1JLT7EKMPBoSTBGykClBMsSk1PrUjLzAFGAUxagoNHSYR3C0iat7ggMbc4Mx0id YpRl2PLghtrmYRY8vLzUqXEeaeAFAmAFGWU5sGNgEXkJUZZKWFeRqCjhHgKUotyM0tQ5V8xin MwKgnz7gSZwpOZVwK36RXQEUxAR9hMjwc5oiQRISXVwBjmKbD1qvkSv/nTT18wUdjysMh2uZH U0aVbzmoIOc9K4dk4S+6o1+E9a5fV8WU+DpswiX/v0+V9S0TYtA6lvV6Y73XlH5tZW/rczXfK giqnf5s31Yc397itwqG2LS5q6++v2/uqWnuThqGs/42fF3VO/+GU8W9+XxgQ/4Hh3VGZ/G+FM xsOLryoxFKckWioxVxUnAgAp6Y6/MMCAAA= X-Env-Sender: JBeulich@suse.com X-Msg-Ref: server-13.tower-206.messagelabs.com!1465914795!35621830!1 X-Originating-IP: [137.65.248.74] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.46; banners=-,-,- X-VirusChecked: Checked Received: (qmail 64253 invoked from network); 14 Jun 2016 14:33:17 -0000 Received: from prv-mh.provo.novell.com (HELO prv-mh.provo.novell.com) (137.65.248.74) by server-13.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 14 Jun 2016 14:33:17 -0000 Received: from INET-PRV-MTA by prv-mh.provo.novell.com with Novell_GroupWise; Tue, 14 Jun 2016 08:33:14 -0600 Message-Id: <576031C802000078000F4D9F@prv-mh.provo.novell.com> X-Mailer: Novell GroupWise Internet Agent 14.2.0 Date: Tue, 14 Jun 2016 08:33:12 -0600 From: "Jan Beulich" To: "xen-devel" Mime-Version: 1.0 Cc: Andrew Cooper Subject: [Xen-devel] [PATCH] x86: show remote CPU state upon fatal NMI X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Quite frequently the watchdog would hit an innocent CPU, e.g. one trying to acquire a spin lock a remote CPU holds for extended periods of time, or a random CPU in TSC calbration rendezvous. In such cases the register and stack dump for that CPU doesn't really help in the analysis of the problem. To keep things reasonable on large systems, only log CS:RIP by default. This can be overridden via a new extension to the "nmi=" command line option such that full register/stack state will get dumped. Signed-off-by: Jan Beulich x86: show remote CPU state upon fatal NMI Quite frequently the watchdog would hit an innocent CPU, e.g. one trying to acquire a spin lock a remote CPU holds for extended periods of time, or a random CPU in TSC calbration rendezvous. In such cases the register and stack dump for that CPU doesn't really help in the analysis of the problem. To keep things reasonable on large systems, only log CS:RIP by default. This can be overridden via a new extension to the "nmi=" command line option such that full register/stack state will get dumped. Signed-off-by: Jan Beulich --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -1155,7 +1155,7 @@ Use the MWAIT idle driver (with model sp of the ACPI based one. ### nmi -> `= ignore | dom0 | fatal` +> `= ignore | dom0 | fatal [,show-all]` > Default: `fatal` for a debug build, or `dom0` for a non-debug build @@ -1163,6 +1163,9 @@ Specify what Xen should do in the event `ignore` discards the error; `dom0` causes Xen to report the error to dom0, while 'fatal' causes Xen to print diagnostics and then hang. +The `show-all` modifier forces all CPUs' full state to be dumped upon +fatal NMIs (normally a result of the watchdog kicking in). + ### noapic Instruct Xen to ignore any IOAPICs that are present in the system, and --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -84,10 +84,11 @@ * dom0: The NMI is virtualised to DOM0. * ignore: The NMI error is cleared and ignored. */ +static char __read_mostly opt_nmi[16] = #ifdef NDEBUG -static char __read_mostly opt_nmi[10] = "dom0"; + "dom0"; #else -static char __read_mostly opt_nmi[10] = "fatal"; + "fatal"; #endif string_param("nmi", opt_nmi); @@ -525,6 +526,35 @@ void vcpu_show_execution_state(struct vc vcpu_unpause(v); } +static cpumask_t nmi_show_state_mask; +static bool_t opt_nmi_show_all; + +static int __init get_nmi_show_all(void) +{ + const char *s = strchr(opt_nmi, ','); + + if ( s && !strcmp(s + 1, "show-all") ) + opt_nmi_show_all = 1; + + return 0; +} +presmp_initcall(get_nmi_show_all); + +static int nmi_show_execution_state(const struct cpu_user_regs *regs, int cpu) +{ + if ( !cpumask_test_cpu(cpu, &nmi_show_state_mask) ) + return 0; + + if ( opt_nmi_show_all ) + show_execution_state(regs); + else + printk(XENLOG_ERR "CPU%d @ %04x:%08lx (%pS)\n", cpu, regs->cs, regs->rip, + guest_mode(regs) ? _p(regs->rip) : NULL); + cpumask_clear_cpu(cpu, &nmi_show_state_mask); + + return 1; +} + static const char *trapstr(unsigned int trapnr) { static const char * const strings[] = { @@ -570,6 +600,15 @@ void fatal_trap(const struct cpu_user_re printk("Faulting linear address: %p\n", _p(cr2)); show_page_walk(cr2); } + else if ( trapnr == TRAP_nmi ) + { + cpumask_andnot(&nmi_show_state_mask, &cpu_online_map, + cpumask_of(smp_processor_id())); + set_nmi_callback(nmi_show_execution_state); + smp_send_nmi_allbutself(); + while ( !cpumask_empty(&nmi_show_state_mask) ) + cpu_relax(); + } } panic("FATAL TRAP: vector = %d (%s)\n" --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -1155,7 +1155,7 @@ Use the MWAIT idle driver (with model sp of the ACPI based one. ### nmi -> `= ignore | dom0 | fatal` +> `= ignore | dom0 | fatal [,show-all]` > Default: `fatal` for a debug build, or `dom0` for a non-debug build @@ -1163,6 +1163,9 @@ Specify what Xen should do in the event `ignore` discards the error; `dom0` causes Xen to report the error to dom0, while 'fatal' causes Xen to print diagnostics and then hang. +The `show-all` modifier forces all CPUs' full state to be dumped upon +fatal NMIs (normally a result of the watchdog kicking in). + ### noapic Instruct Xen to ignore any IOAPICs that are present in the system, and --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -84,10 +84,11 @@ * dom0: The NMI is virtualised to DOM0. * ignore: The NMI error is cleared and ignored. */ +static char __read_mostly opt_nmi[16] = #ifdef NDEBUG -static char __read_mostly opt_nmi[10] = "dom0"; + "dom0"; #else -static char __read_mostly opt_nmi[10] = "fatal"; + "fatal"; #endif string_param("nmi", opt_nmi); @@ -525,6 +526,35 @@ void vcpu_show_execution_state(struct vc vcpu_unpause(v); } +static cpumask_t nmi_show_state_mask; +static bool_t opt_nmi_show_all; + +static int __init get_nmi_show_all(void) +{ + const char *s = strchr(opt_nmi, ','); + + if ( s && !strcmp(s + 1, "show-all") ) + opt_nmi_show_all = 1; + + return 0; +} +presmp_initcall(get_nmi_show_all); + +static int nmi_show_execution_state(const struct cpu_user_regs *regs, int cpu) +{ + if ( !cpumask_test_cpu(cpu, &nmi_show_state_mask) ) + return 0; + + if ( opt_nmi_show_all ) + show_execution_state(regs); + else + printk(XENLOG_ERR "CPU%d @ %04x:%08lx (%pS)\n", cpu, regs->cs, regs->rip, + guest_mode(regs) ? _p(regs->rip) : NULL); + cpumask_clear_cpu(cpu, &nmi_show_state_mask); + + return 1; +} + static const char *trapstr(unsigned int trapnr) { static const char * const strings[] = { @@ -570,6 +600,15 @@ void fatal_trap(const struct cpu_user_re printk("Faulting linear address: %p\n", _p(cr2)); show_page_walk(cr2); } + else if ( trapnr == TRAP_nmi ) + { + cpumask_andnot(&nmi_show_state_mask, &cpu_online_map, + cpumask_of(smp_processor_id())); + set_nmi_callback(nmi_show_execution_state); + smp_send_nmi_allbutself(); + while ( !cpumask_empty(&nmi_show_state_mask) ) + cpu_relax(); + } } panic("FATAL TRAP: vector = %d (%s)\n"