From patchwork Mon Apr 17 07:47:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 9683633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7C824602C2 for ; Mon, 17 Apr 2017 07:50:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B4192684F for ; Mon, 17 Apr 2017 07:50:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6014F26BE9; Mon, 17 Apr 2017 07:50:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7900126E64 for ; Mon, 17 Apr 2017 07:50:28 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d01OA-0005vR-I7; Mon, 17 Apr 2017 07:47:58 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d01O8-0005vK-Tf for xen-devel@lists.xenproject.org; Mon, 17 Apr 2017 07:47:57 +0000 Received: from [85.158.139.211] by server-3.bemta-5.messagelabs.com id 84/67-01936-C2374F85; Mon, 17 Apr 2017 07:47:56 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrLIsWRWlGSWpSXmKPExsWyU9JRQle7+Eu EwcMOGYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNeP3lolMBZssKmauX8TewPhXr4uRk0NCwE/i zYVNTCA2i4CqRNexXyxdjBwcbAL2EtO/VoCERQTcJSauWMEKYgsLmEg8nfGVBcTmFfCUeN1wD axVSKBcom3+f2aIuKDEyZlPwGqYBfQkbkydwgYykllAWmL5Pw6IsLxE89bZYOWcApYSq67dBi sXFVCR2Pj6FTPESEWJ/nkP2CCuTJeY+KyHZQIj/ywkG2Yh2TALYcMsJBsWMLKsYlQvTi0qSy3 SNdRLKspMzyjJTczM0TU0MNXLTS0uTkxPzUlMKtZLzs/dxAgMynoGBsYdjE29zocYJTmYlER5 xVS/RAjxJeWnVGYkFmfEF5XmpBYfYtTg4BDYvHb1BUYplrz8vFQlCd7LhUB1gkWp6akVaZk5w LiBKZXg4FES4S0HSfMWFyTmFmemQ6ROMSpKifMqFgElBEASGaV5cG2wWL3EKCslzMvIwMAgxF OQWpSbWYIq/4pRnINRSZhXF2QKT2ZeCdz0V0CLmYAWRwSALS5JREhJNTAm9VaZ384/lH60rn7 bDzXer+YH9pzbG5yxIsbo5ZXtb7j8/94+2LnOdq9K92Feq5wL56MeqadLMX43UP/AqHH80WZ1 xyUiq90yvyq8fLROlt32zsr/mkmTVx902LB9/k+nuFb9LZsrTBb/4DfQ+OmSezM+PbfwdvnbF NVcZ53pUvHnore78NoosRRnJBpqMRcVJwIANyOoGtACAAA= X-Env-Sender: prvs=2732fb06d=roger.pau@citrix.com X-Msg-Ref: server-14.tower-206.messagelabs.com!1492415275!54847577!1 X-Originating-IP: [185.25.65.24] X-SpamReason: No, hits=0.0 required=7.0 tests=received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 23160 invoked from network); 17 Apr 2017 07:47:55 -0000 Received: from smtp.ctxuk.citrix.com (HELO SMTP.EU.CITRIX.COM) (185.25.65.24) by server-14.tower-206.messagelabs.com with RC4-SHA encrypted SMTP; 17 Apr 2017 07:47:55 -0000 X-IronPort-AV: E=Sophos;i="5.37,213,1488844800"; d="scan'208";a="44413113" Date: Mon, 17 Apr 2017 08:47:48 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: , Kevin Tian Message-ID: <20170417074748.335yjh6dtgp5mi3b@dhcp-3-128.uk.xensource.com> References: <20170414153441.nd76764ufwq5uwpc@dhcp-3-128.uk.xensource.com> <20170416233245.GA5141@skl-2s3.sh.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20170416233245.GA5141@skl-2s3.sh.intel.com> User-Agent: NeoMutt/20170306 (1.8.0) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) Subject: Re: [Xen-devel] PVH Dom0 Intel IOMMU issues X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Apr 17, 2017 at 07:32:45AM +0800, Chao Gao wrote: > On Fri, Apr 14, 2017 at 04:34:41PM +0100, Roger Pau Monné wrote: > >Hello, > > > >Although PVHv2 Dom0 is not yet finished, I've been trying the current code on > >different hardware, and found that with pre-Haswell Intel hardware PVHv2 Dom0 > >completely freezes the box when calling iommu_hwdom_init in dom0_construct_pvh. > >OTOH the same doesn't happen when using a newer CPU (ie: haswell or newer). > > > >I'm not able to debug that in any meaningful way because the box seems to lock > >up completely, even the watchdog NMI stops working. Here is the boot log, up to > >the point where it freezes: > > I try "dom0=pvh" with my skylake. An assertion failed. Is it a software bug? > [...] > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) > (XEN) [ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) > (XEN) [ 0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-31 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec08000] gsi_base[32]) > (XEN) [ 0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec08000, GSI 32-39 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x0b] address[0xfec10000] gsi_base[40]) > (XEN) [ 0.000000] IOAPIC[3]: apic_id 11, version 32, address 0xfec10000, GSI 40-47 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x0c] address[0xfec18000] gsi_base[48]) > (XEN) [ 0.000000] IOAPIC[4]: apic_id 12, version 32, address 0xfec18000, GSI 48-55 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x0f] address[0xfec20000] gsi_base[72]) > (XEN) [ 0.000000] IOAPIC[5]: apic_id 15, version 32, address 0xfec20000, GSI 72-79 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x10] address[0xfec28000] gsi_base[80]) > (XEN) [ 0.000000] IOAPIC[6]: apic_id 16, version 32, address 0xfec28000, GSI 80-87 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x11] address[0xfec30000] gsi_base[88]) > (XEN) [ 0.000000] IOAPIC[7]: apic_id 17, version 32, address 0xfec30000, GSI 88-95 > (XEN) [ 0.000000] ACPI: IOAPIC (id[0x12] address[0xfec38000] gsi_base[96]) > (XEN) [ 0.000000] IOAPIC[8]: apic_id 18, version 32, address 0xfec38000, GSI 96-103 [...] > (XEN) [ 0.000000] IRQ limits: 104 GSI, 21416 MSI/MSI-X [...] > (XEN) [ 14.147217] Dom0 has maximum 1448 PIRQs > (XEN) [ 14.151527] Assertion 'hvm_domain_irq(d)->nr_gsis == nr_gsis' failed at vioapic.c:600 > (XEN) [ 14.154404] ----[ Xen-4.9-unstable x86_64 debug=y Not tainted ]---- > (XEN) [ 14.155867] CPU: 0 > (XEN) [ 14.157286] RIP: e008:[] vioapic_init+0x110/0x167 > (XEN) [ 14.158750] RFLAGS: 0000000000010287 CONTEXT: hypervisor > (XEN) [ 14.160203] rax: ffff830837c7fa00 rbx: 0000000000000009 rcx: 00000000c8381c70 > (XEN) [ 14.163073] rdx: 0000000000000071 rsi: ffff830837c7e400 rdi: ffff83083fff7868 > (XEN) [ 14.165937] rbp: ffff82d080457d28 rsp: ffff82d080457ce8 r8: ffff82e000000000 > (XEN) [ 14.168797] r9: 0000000000000381 r10: ffff82d08045f400 r11: 0000000000000000 > (XEN) [ 14.171657] r12: 0000000000000008 r13: ffff830837d29000 r14: 0000000000000058 > (XEN) [ 14.174568] r15: ffff830837c7fb20 cr0: 000000008005003b cr4: 00000000003526e0 > (XEN) [ 14.177437] cr3: 000000006f84c000 cr2: 0000000000000000 > (XEN) [ 14.178887] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 > (XEN) [ 14.181753] Xen code around (vioapic_init+0x110/0x167): > (XEN) [ 14.184609] 00 00 44 3b 70 40 74 02 <0f> 0b 8b 45 cc 41 89 85 b0 02 00 00 4c 89 ef e8 > (XEN) [ 14.187473] Xen stack trace from rsp=ffff82d080457ce8: > (XEN) [ 14.188916] ffff82d08029e7de 0000000937c7f010 ffff82d080457d08 ffff830837d29000 > (XEN) [ 14.191784] 0000000000000068 0000000000000001 0000000000000000 0000000000000000 > (XEN) [ 14.194645] ffff82d080457d48 ffff82d0802de276 ffff830837d29000 0000000000000000 > (XEN) [ 14.197507] ffff82d080457d78 ffff82d08026d593 ffff82d080457d78 ffff830837d29000 > (XEN) [ 14.200371] 000000000000001f 0000000000000007 ffff82d080457de8 ffff82d080205226 > (XEN) [ 14.203234] ffff82d0804380e0 0000000000000004 ffff82d080457eb4 0000000000000000 > (XEN) [ 14.206097] ffff82d080457dc8 f7fa32231fcbfbff 000001212c100800 00000000000000e0 > (XEN) [ 14.208956] ffff830838543850 00000000000000e0 ffff82d08043b780 000000000000006f > (XEN) [ 14.211817] ffff82d080457f08 ffff82d0803ee1be 000000000028fe80 000000000000015c > (XEN) [ 14.214739] 00000000000001df 0000000200000000 0000000000000002 0000000000000002 > (XEN) [ 14.217598] 0000000000000002 0000000000000001 0000000000000001 0000000000000001 > (XEN) [ 14.220459] 0000000000000001 0000000000000000 ffff82d080429a90 0000000000000017 > (XEN) [ 14.223317] 0000001075ec7000 00000000013b7000 0000000001080000 0000000000000000 > (XEN) [ 14.226178] ffff830000000002 ffff830000097c60 ffff830000097ee0 ffff830000097fb0 > (XEN) [ 14.229039] 0000000000000000 0000001100000000 0000000800000000 000000010000006e > (XEN) [ 14.231901] 0000000000000003 00000000000002f8 0000000000000000 0000000000000000 > (XEN) [ 14.234770] 00000000000000e8 0000000068924a50 000000006c5b8a28 000000006c5b8a40 > (XEN) [ 14.237640] 0000000000000000 ffff82d0802000f3 0000000000000000 0000000000000000 > (XEN) [ 14.240504] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) [ 14.243374] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) [ 14.246245] Xen call trace: > (XEN) [ 14.247668] [] vioapic_init+0x110/0x167 > (XEN) [ 14.249126] [] hvm_domain_initialise+0x268/0x365 > (XEN) [ 14.250593] [] arch_domain_create+0x4b0/0x6ac > (XEN) [ 14.252055] [] domain_create+0x3d8/0x593 > (XEN) [ 14.253578] [] __start_xen+0x20fa/0x24c3 > (XEN) [ 14.255029] [] __high_start+0x53/0x58 > (XEN) [ 14.256481] > (XEN) [ 14.510271] > (XEN) [ 14.513527] **************************************** > (XEN) [ 14.520279] Panic on CPU 0: > (XEN) [ 14.524726] Assertion 'hvm_domain_irq(d)->nr_gsis == nr_gsis' failed at vioapic.c:600 > (XEN) [ 14.535470] **************************************** > (XEN) [ 14.542091] > (XEN) [ 14.545230] Reboot in five seconds... > (XEN) [ 19.553630] Resetting with ACPI MEMORY or I/O RESET_REG. This is certainly unexpected. Can you try again with the debug patch below applied? It should show what's actually wrong. Thanks, Roger. ---8<--- diff --git a/xen/arch/x86/hvm/vioapic.c b/xen/arch/x86/hvm/vioapic.c index 527ac2aadd..1df7710041 100644 --- a/xen/arch/x86/hvm/vioapic.c +++ b/xen/arch/x86/hvm/vioapic.c @@ -625,6 +625,9 @@ int vioapic_init(struct domain *d) nr_gsis += nr_pins; } + printk("domain nr_gsis: %u vioapic gsis: %u nr_irqs_gsi: %u highest_gsi: %u\n", + hvm_domain_irq(d)->nr_gsis, nr_gsis, nr_irqs_gsi, highest_gsi()); + ASSERT(hvm_domain_irq(d)->nr_gsis == nr_gsis); d->arch.hvm_domain.nr_vioapics = nr_vioapics;