From patchwork Thu Jun 16 11:26:46 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 9180535 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 28FD26075D for ; Thu, 16 Jun 2016 11:29:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18D8B27F07 for ; Thu, 16 Jun 2016 11:29:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D92C28308; Thu, 16 Jun 2016 11:29:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EEE8B27F07 for ; Thu, 16 Jun 2016 11:28:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bDVRj-0000b1-V0; Thu, 16 Jun 2016 11:26:51 +0000 Received: from mail6.bemta6.messagelabs.com ([85.158.143.247]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bDVRi-0000aq-2l for xen-devel@lists.xenproject.org; Thu, 16 Jun 2016 11:26:50 +0000 Received: from [85.158.143.35] by server-1.bemta-6.messagelabs.com id 50/DE-09256-9FC82675; Thu, 16 Jun 2016 11:26:49 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrEIsWRWlGSWpSXmKPExsXS6fjDS/dHT1K 4wfU9Ehbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8a/lWvYCvpTK1Y33GVrYOwJ6GLk5BASyJN4 dnoHUxcjBwevgJ3Ew6v6IGEJAUOJffNXsYHYLAKqEuuvXGUHsdkE1CXanm1nBSkXETCQOHc0C cRkFtCX2LaOBaRCWMBX4v273ywQAwUl/u4QBgkzA83+9/s/0wRGrlkImVlIMhC2lsTDX7dYIG xtiWULXzPPApsvLbH8HwdE2EHi5LVtaEpAbG+JWztuMi1g5FjFqF6cWlSWWqRrqJdUlJmeUZK bmJmja2hgppebWlycmJ6ak5hUrJecn7uJERh2DECwg3Hnc6dDjJIcTEqivPX1SeFCfEn5KZUZ icUZ8UWlOanFhxhlODiUJHjndQPlBItS01Mr0jJzgBEAk5bg4FES4WUASfMWFyTmFmemQ6ROM SpKifOuBEkIgCQySvPg2mBRd4lRVkqYlxHoECGegtSi3MwSVPlXjOIcjErCvEtBpvBk5pXATX 8FtJgJaLHN9HiQxSWJCCmpBkYPnl8vPIt551q9ltNtVg9exrLbWlpklmRrMV9yZWX+SyVlFm3 NE1mRV01bag5ODGo5W85Udyht6etmzuvNLJ9Wbzc2mP88PWD9ShnXYiHFgxIODmtqZNUNohZs ENgxtV9m261goTNLtp5n071te1vy9osHzdN7NaZV2MgoPPkcz1ls8EVUR4mlOCPRUIu5qDgRA DCdjBC1AgAA X-Env-Sender: JBeulich@suse.com X-Msg-Ref: server-12.tower-21.messagelabs.com!1466076406!19373417!1 X-Originating-IP: [137.65.248.74] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.46; banners=-,-,- X-VirusChecked: Checked Received: (qmail 3463 invoked from network); 16 Jun 2016 11:26:47 -0000 Received: from prv-mh.provo.novell.com (HELO prv-mh.provo.novell.com) (137.65.248.74) by server-12.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 16 Jun 2016 11:26:47 -0000 Received: from INET-PRV-MTA by prv-mh.provo.novell.com with Novell_GroupWise; Thu, 16 Jun 2016 05:26:45 -0600 Message-Id: <5762A91602000078000F5A71@prv-mh.provo.novell.com> X-Mailer: Novell GroupWise Internet Agent 14.2.0 Date: Thu, 16 Jun 2016 05:26:46 -0600 From: "Jan Beulich" To: "xen-devel" Mime-Version: 1.0 Cc: Andrew Cooper Subject: [Xen-devel] [PATCH v2] x86: show remote CPU state upon fatal NMI or unknown MCE X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Quite frequently the watchdog would hit an innocent CPU, e.g. one trying to acquire a spin lock a remote CPU holds for extended periods of time, or a random CPU in TSC calbration rendezvous. In such cases the register and stack dump for that CPU doesn't really help in the analysis of the problem. To keep things reasonable on large systems, only log CS:RIP by default. This can be overridden via a new command line option such that full register/stack state would get logged. Signed-off-by: Jan Beulich --- v2: Pass flag to fatal_trap(). x86: show remote CPU state upon fatal NMI or unknown MCE Quite frequently the watchdog would hit an innocent CPU, e.g. one trying to acquire a spin lock a remote CPU holds for extended periods of time, or a random CPU in TSC calbration rendezvous. In such cases the register and stack dump for that CPU doesn't really help in the analysis of the problem. To keep things reasonable on large systems, only log CS:RIP by default. This can be overridden via a new command line option such that full register/stack state would get logged. Signed-off-by: Jan Beulich --- v2: Pass flag to fatal_trap(). --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -198,6 +198,14 @@ Permit Xen to use Address Space Identifi tags the TLB entries with an ID per vcpu. This allows for guest TLB flushes to be performed without the overhead of a complete TLB flush. +### async-show-all +> `= ` + +> Default: `false` + +Forces all CPUs' full state to be logged upon certain fatal asynchronous +exceptions (watchdog NMIs and unexpected MCEs). + ### ats > `= ` --- a/xen/arch/x86/cpu/mcheck/mce.c +++ b/xen/arch/x86/cpu/mcheck/mce.c @@ -77,7 +77,7 @@ static void unexpected_machine_check(con { console_force_unlock(); printk("Unexpected Machine Check Exception\n"); - fatal_trap(regs); + fatal_trap(regs, 1); } --- a/xen/arch/x86/nmi.c +++ b/xen/arch/x86/nmi.c @@ -488,7 +488,7 @@ bool_t nmi_watchdog_tick(const struct cp console_force_unlock(); printk("Watchdog timer detects that CPU%d is stuck!\n", smp_processor_id()); - fatal_trap(regs); + fatal_trap(regs, 1); } } else --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -525,6 +525,25 @@ void vcpu_show_execution_state(struct vc vcpu_unpause(v); } +static cpumask_t show_state_mask; +static bool_t opt_show_all; +boolean_param("async-show-all", opt_show_all); + +static int nmi_show_execution_state(const struct cpu_user_regs *regs, int cpu) +{ + if ( !cpumask_test_cpu(cpu, &show_state_mask) ) + return 0; + + if ( opt_show_all ) + show_execution_state(regs); + else + printk(XENLOG_ERR "CPU%d @ %04x:%08lx (%pS)\n", cpu, regs->cs, regs->rip, + guest_mode(regs) ? _p(regs->rip) : NULL); + cpumask_clear_cpu(cpu, &show_state_mask); + + return 1; +} + static const char *trapstr(unsigned int trapnr) { static const char * const strings[] = { @@ -544,7 +563,7 @@ static const char *trapstr(unsigned int * are disabled). In such situations we can't do much that is safe. We try to * print out some tracing and then we just spin. */ -void fatal_trap(const struct cpu_user_regs *regs) +void fatal_trap(const struct cpu_user_regs *regs, bool_t show_remote) { static DEFINE_PER_CPU(char, depth); unsigned int trapnr = regs->entry_vector; @@ -570,6 +589,15 @@ void fatal_trap(const struct cpu_user_re printk("Faulting linear address: %p\n", _p(cr2)); show_page_walk(cr2); } + if ( show_remote ) + { + cpumask_andnot(&show_state_mask, &cpu_online_map, + cpumask_of(smp_processor_id())); + set_nmi_callback(nmi_show_execution_state); + smp_send_nmi_allbutself(); + while ( !cpumask_empty(&show_state_mask) ) + cpu_relax(); + } } panic("FATAL TRAP: vector = %d (%s)\n" @@ -1711,7 +1739,7 @@ void do_page_fault(struct cpu_user_regs { console_start_sync(); printk("Xen SM%cP violation\n", (pf_type == smep_fault) ? 'E' : 'A'); - fatal_trap(regs); + fatal_trap(regs, 0); } if ( pf_type != real_fault ) @@ -1782,7 +1810,7 @@ void __init do_early_page_fault(struct c console_start_sync(); printk("Early fatal page fault at %04x:%p (cr2=%p, ec=%04x)\n", regs->cs, _p(regs->eip), _p(cr2), regs->error_code); - fatal_trap(regs); + fatal_trap(regs, 0); } } @@ -3598,7 +3626,7 @@ static void pci_serr_error(const struct default: /* 'fatal' */ console_force_unlock(); printk("\n\nNMI - PCI system error (SERR)\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } } @@ -3613,7 +3641,7 @@ static void io_check_error(const struct default: /* 'fatal' */ console_force_unlock(); printk("\n\nNMI - I/O ERROR\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } outb((inb(0x61) & 0x0f) | 0x08, 0x61); /* clear-and-disable IOCK */ @@ -3633,7 +3661,7 @@ static void unknown_nmi_error(const stru console_force_unlock(); printk("Uhhuh. NMI received for unknown reason %02x.\n", reason); printk("Do you have a strange power saving mode enabled?\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } } --- a/xen/arch/x86/x86_64/entry.S +++ b/xen/arch/x86/x86_64/entry.S @@ -558,6 +558,7 @@ exception_with_ints_disabled: /* No special register assumptions. */ FATAL_exception_with_ints_disabled: + xorl %esi,%esi movq %rsp,%rdi call fatal_trap BUG /* fatal_trap() shouldn't return. */ --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -541,7 +541,7 @@ void show_registers(const struct cpu_use void show_execution_state(const struct cpu_user_regs *regs); #define dump_execution_state() run_in_exception_handler(show_execution_state) void show_page_walk(unsigned long addr); -void noreturn fatal_trap(const struct cpu_user_regs *regs); +void noreturn fatal_trap(const struct cpu_user_regs *regs, bool_t show_remote); void compat_show_guest_stack(struct vcpu *v, const struct cpu_user_regs *regs, int lines); --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -198,6 +198,14 @@ Permit Xen to use Address Space Identifi tags the TLB entries with an ID per vcpu. This allows for guest TLB flushes to be performed without the overhead of a complete TLB flush. +### async-show-all +> `= ` + +> Default: `false` + +Forces all CPUs' full state to be logged upon certain fatal asynchronous +exceptions (watchdog NMIs and unexpected MCEs). + ### ats > `= ` --- a/xen/arch/x86/cpu/mcheck/mce.c +++ b/xen/arch/x86/cpu/mcheck/mce.c @@ -77,7 +77,7 @@ static void unexpected_machine_check(con { console_force_unlock(); printk("Unexpected Machine Check Exception\n"); - fatal_trap(regs); + fatal_trap(regs, 1); } --- a/xen/arch/x86/nmi.c +++ b/xen/arch/x86/nmi.c @@ -488,7 +488,7 @@ bool_t nmi_watchdog_tick(const struct cp console_force_unlock(); printk("Watchdog timer detects that CPU%d is stuck!\n", smp_processor_id()); - fatal_trap(regs); + fatal_trap(regs, 1); } } else --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -525,6 +525,25 @@ void vcpu_show_execution_state(struct vc vcpu_unpause(v); } +static cpumask_t show_state_mask; +static bool_t opt_show_all; +boolean_param("async-show-all", opt_show_all); + +static int nmi_show_execution_state(const struct cpu_user_regs *regs, int cpu) +{ + if ( !cpumask_test_cpu(cpu, &show_state_mask) ) + return 0; + + if ( opt_show_all ) + show_execution_state(regs); + else + printk(XENLOG_ERR "CPU%d @ %04x:%08lx (%pS)\n", cpu, regs->cs, regs->rip, + guest_mode(regs) ? _p(regs->rip) : NULL); + cpumask_clear_cpu(cpu, &show_state_mask); + + return 1; +} + static const char *trapstr(unsigned int trapnr) { static const char * const strings[] = { @@ -544,7 +563,7 @@ static const char *trapstr(unsigned int * are disabled). In such situations we can't do much that is safe. We try to * print out some tracing and then we just spin. */ -void fatal_trap(const struct cpu_user_regs *regs) +void fatal_trap(const struct cpu_user_regs *regs, bool_t show_remote) { static DEFINE_PER_CPU(char, depth); unsigned int trapnr = regs->entry_vector; @@ -570,6 +589,15 @@ void fatal_trap(const struct cpu_user_re printk("Faulting linear address: %p\n", _p(cr2)); show_page_walk(cr2); } + if ( show_remote ) + { + cpumask_andnot(&show_state_mask, &cpu_online_map, + cpumask_of(smp_processor_id())); + set_nmi_callback(nmi_show_execution_state); + smp_send_nmi_allbutself(); + while ( !cpumask_empty(&show_state_mask) ) + cpu_relax(); + } } panic("FATAL TRAP: vector = %d (%s)\n" @@ -1711,7 +1739,7 @@ void do_page_fault(struct cpu_user_regs { console_start_sync(); printk("Xen SM%cP violation\n", (pf_type == smep_fault) ? 'E' : 'A'); - fatal_trap(regs); + fatal_trap(regs, 0); } if ( pf_type != real_fault ) @@ -1782,7 +1810,7 @@ void __init do_early_page_fault(struct c console_start_sync(); printk("Early fatal page fault at %04x:%p (cr2=%p, ec=%04x)\n", regs->cs, _p(regs->eip), _p(cr2), regs->error_code); - fatal_trap(regs); + fatal_trap(regs, 0); } } @@ -3598,7 +3626,7 @@ static void pci_serr_error(const struct default: /* 'fatal' */ console_force_unlock(); printk("\n\nNMI - PCI system error (SERR)\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } } @@ -3613,7 +3641,7 @@ static void io_check_error(const struct default: /* 'fatal' */ console_force_unlock(); printk("\n\nNMI - I/O ERROR\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } outb((inb(0x61) & 0x0f) | 0x08, 0x61); /* clear-and-disable IOCK */ @@ -3633,7 +3661,7 @@ static void unknown_nmi_error(const stru console_force_unlock(); printk("Uhhuh. NMI received for unknown reason %02x.\n", reason); printk("Do you have a strange power saving mode enabled?\n"); - fatal_trap(regs); + fatal_trap(regs, 0); } } --- a/xen/arch/x86/x86_64/entry.S +++ b/xen/arch/x86/x86_64/entry.S @@ -558,6 +558,7 @@ exception_with_ints_disabled: /* No special register assumptions. */ FATAL_exception_with_ints_disabled: + xorl %esi,%esi movq %rsp,%rdi call fatal_trap BUG /* fatal_trap() shouldn't return. */ --- a/xen/include/asm-x86/processor.h +++ b/xen/include/asm-x86/processor.h @@ -541,7 +541,7 @@ void show_registers(const struct cpu_use void show_execution_state(const struct cpu_user_regs *regs); #define dump_execution_state() run_in_exception_handler(show_execution_state) void show_page_walk(unsigned long addr); -void noreturn fatal_trap(const struct cpu_user_regs *regs); +void noreturn fatal_trap(const struct cpu_user_regs *regs, bool_t show_remote); void compat_show_guest_stack(struct vcpu *v, const struct cpu_user_regs *regs, int lines);