From patchwork Wed May 29 10:17:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jan Beulich X-Patchwork-Id: 10966709 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB80A15E6 for ; Wed, 29 May 2019 10:20:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB51E28644 for ; Wed, 29 May 2019 10:20:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A47D928960; Wed, 29 May 2019 10:20:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4850F28644 for ; Wed, 29 May 2019 10:20:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hVvef-0003IO-G9; Wed, 29 May 2019 10:17:57 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hVvee-0003IJ-St for xen-devel@lists.xenproject.org; Wed, 29 May 2019 10:17:56 +0000 X-Inumbo-ID: 048c7eec-81fb-11e9-9ea5-abe0f3483738 Received: from prv1-mh.provo.novell.com (unknown [137.65.248.33]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 048c7eec-81fb-11e9-9ea5-abe0f3483738; Wed, 29 May 2019 10:17:53 +0000 (UTC) Received: from INET-PRV1-MTA by prv1-mh.provo.novell.com with Novell_GroupWise; Wed, 29 May 2019 04:17:52 -0600 Message-Id: <5CEE5C4D02000078002335A0@prv1-mh.provo.novell.com> X-Mailer: Novell GroupWise Internet Agent 18.1.1 Date: Wed, 29 May 2019 04:17:49 -0600 From: "Jan Beulich" To: "xen-devel" Mime-Version: 1.0 Content-Disposition: inline Subject: [Xen-devel] [PATCH] x86/SMP: don't try to stop already stopped CPUs X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP In particular with an enabled IOMMU (but not really limited to this case), trying to invoke fixup_irqs() after having already done disable_IO_APIC() -> clear_IO_APIC() is a rather bad idea: RIP: e008:[] amd_iommu_read_ioapic_from_ire+0xde/0x113 RFLAGS: 0000000000010006 CONTEXT: hypervisor (d0v0) rax: ffff8320291de00c rbx: 0000000000000003 rcx: ffff832035000000 rdx: 0000000000000000 rsi: 0000000000000000 rdi: ffff82d0805ca840 rbp: ffff83009e8a79c8 rsp: ffff83009e8a79a8 r8: 0000000000000000 r9: 0000000000000004 r10: 000000000008b9f9 r11: 0000000000000006 r12: 0000000000010000 r13: 0000000000000003 r14: 0000000000000000 r15: 00000000fffeffff cr0: 0000000080050033 cr4: 00000000003406e0 cr3: 0000002035d59000 cr2: ffff88824ccb4ee0 fsb: 00007f2143f08840 gsb: ffff888256a00000 gss: 0000000000000000 ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 Xen code around (amd_iommu_read_ioapic_from_ire+0xde/0x113): ff 07 00 00 39 d3 74 02 <0f> 0b 41 81 e4 00 f8 ff ff 8b 10 89 d0 25 00 00 Xen stack trace from rsp=ffff83009e8a79a8: ... Xen call trace: [] amd_iommu_read_ioapic_from_ire+0xde/0x113 [] iommu_read_apic_from_ire+0x10/0x12 [] io_apic.c#modify_IO_APIC_irq+0x5e/0x126 [] io_apic.c#unmask_IO_APIC_irq+0x2d/0x41 [] fixup_irqs+0x320/0x40b [] smp_send_stop+0x4b/0xa8 [] machine_restart+0x98/0x288 [] console_suspend+0/0x28 [] do_general_protection+0x204/0x24e [] x86_64/entry.S#handle_exception_saved+0x68/0x94 [<00000000aa5b526b>] 00000000aa5b526b [] machine_restart+0x1e6/0x288 [] hwdom_shutdown+0xa2/0x11d [] domain_shutdown+0x4f/0xd8 [] do_sched_op+0x12f/0x42a [] pv_hypercall+0x1e4/0x564 [] lstar_enter+0x112/0x120 Don't call fixup_irqs() and don't send any IPI if there's only one online CPU anyway, and don't call __stop_this_cpu() at all when the CPU we're on was already marked offline (by a prior invocation of __stop_this_cpu()). Reported-by: Andrew Cooper Signed-off-by: Jan Beulich Reviewed-by: Roger Pau Monné Reported-by: Andrew Cooper <andrew.cooper3@citrix.com> Signed-off-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Andrew Cooper Reported-by: Andrew Cooper <andrew.cooper3@citrix.com> Signed-off-by: Jan Beulich <jbeulich@suse.com> --- a/xen/arch/x86/smp.c +++ b/xen/arch/x86/smp.c @@ -302,23 +302,31 @@ static void stop_this_cpu(void *dummy) */ void smp_send_stop(void) { - int timeout = 10; + unsigned int cpu = smp_processor_id(); - local_irq_disable(); - fixup_irqs(cpumask_of(smp_processor_id()), 0); - local_irq_enable(); - - smp_call_function(stop_this_cpu, NULL, 0); - - /* Wait 10ms for all other CPUs to go offline. */ - while ( (num_online_cpus() > 1) && (timeout-- > 0) ) - mdelay(1); - - local_irq_disable(); - disable_IO_APIC(); - hpet_disable(); - __stop_this_cpu(); - local_irq_enable(); + if ( num_online_cpus() > 1 ) + { + int timeout = 10; + + local_irq_disable(); + fixup_irqs(cpumask_of(cpu), 0); + local_irq_enable(); + + smp_call_function(stop_this_cpu, NULL, 0); + + /* Wait 10ms for all other CPUs to go offline. */ + while ( (num_online_cpus() > 1) && (timeout-- > 0) ) + mdelay(1); + } + + if ( cpu_online(cpu) ) + { + local_irq_disable(); + disable_IO_APIC(); + hpet_disable(); + __stop_this_cpu(); + local_irq_enable(); + } } void smp_send_nmi_allbutself(void)