From patchwork Fri Jan 9 21:15:58 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konrad Rzeszutek Wilk X-Patchwork-Id: 5602941 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 8825D9F1C5 for ; Fri, 9 Jan 2015 21:19:15 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7269620634 for ; Fri, 9 Jan 2015 21:19:14 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 562CB205DA for ; Fri, 9 Jan 2015 21:19:13 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Y9gvO-0005uN-QZ; Fri, 09 Jan 2015 21:16:54 +0000 Received: from userp1040.oracle.com ([156.151.31.81]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1Y9gvL-0005t9-4g for linux-arm-kernel@lists.infradead.org; Fri, 09 Jan 2015 21:16:51 +0000 Received: from acsinet22.oracle.com (acsinet22.oracle.com [141.146.126.238]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id t09LG5u0003050 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 9 Jan 2015 21:16:06 GMT Received: from userz7021.oracle.com (userz7021.oracle.com [156.151.31.85]) by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id t09LG2nU025138 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 9 Jan 2015 21:16:03 GMT Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by userz7021.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id t09LG1tU011059; Fri, 9 Jan 2015 21:16:01 GMT Received: from l.oracle.com (/10.137.179.11) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 09 Jan 2015 13:16:01 -0800 Received: by l.oracle.com (Postfix, from userid 1000) id 151096A0537; Fri, 9 Jan 2015 16:15:58 -0500 (EST) Date: Fri, 9 Jan 2015 16:15:58 -0500 From: Konrad Rzeszutek Wilk To: Jiang Liu Subject: Re: [Bugfix] x86/apic: Fix xen IRQ allocation failure caused by commit b81975eade8c Message-ID: <20150109211557.GA4751@l.oracle.com> References: <1420611244-25624-1-git-send-email-jiang.liu@linux.intel.com> <20150107145050.GC30457@l.oracle.com> <54AD52D0.5050303@linux.intel.com> <20150107154416.GB30955@l.oracle.com> <54AE2576.5020209@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <54AE2576.5020209@linux.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Source-IP: acsinet22.oracle.com [141.146.126.238] X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20150109_131651_246045_F4C0CF05 X-CRM114-Status: GOOD ( 25.53 ) X-Spam-Score: -2.3 (--) Cc: Prarit Bhargava , Tony Luck , xen-devel@lists.xenproject.org, Ingo Molnar , Richard Weinberger , x86@kernel.org, linux-pci@vger.kernel.org, Oren Twaig , linux-kernel@vger.kernel.org, Sander Eikelenboom , HATAYAMA Daisuke , Ingo Molnar , David Vrabel , Jan Beulich , "H. Peter Anvin" , Grant Likely , David Rientjes , Boris Ostrovsky , Yinghai Lu , Thomas Gleixner , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Jan 08, 2015 at 02:36:38PM +0800, Jiang Liu wrote: > On 2015/1/7 23:44, Konrad Rzeszutek Wilk wrote: > > On Wed, Jan 07, 2015 at 11:37:52PM +0800, Jiang Liu wrote: > >> On 2015/1/7 22:50, Konrad Rzeszutek Wilk wrote: > >>> On Wed, Jan 07, 2015 at 02:13:49PM +0800, Jiang Liu wrote: > >>>> Commit b81975eade8c ("x86, irq: Clean up irqdomain transition code") > >>>> breaks xen IRQ allocation because xen_smp_prepare_cpus() doesn't invoke > >>>> setup_IO_APIC(), so no irqdomains created for IOAPICs and > >>>> mp_map_pin_to_irq() fails at the very beginning. > >>>> --- a/arch/x86/kernel/apic/io_apic.c > >>>> +++ b/arch/x86/kernel/apic/io_apic.c > >>>> @@ -2369,31 +2369,29 @@ static void ioapic_destroy_irqdomain(int idx) > >>>> ioapics[idx].pin_info = NULL; > >>>> } > >>>> > >>>> -void __init setup_IO_APIC(void) > >>>> +void __init setup_IO_APIC(bool xen_smp) > >>>> { > >>>> int ioapic; > >>>> > >>>> - /* > >>>> - * calling enable_IO_APIC() is moved to setup_local_APIC for BP > >>>> - */ > >>>> - io_apic_irqs = nr_legacy_irqs() ? ~PIC_IRQS : ~0UL; > >>>> + if (!xen_smp) { > >>>> + apic_printk(APIC_VERBOSE, "ENABLING IO-APIC IRQs\n"); > >>>> + io_apic_irqs = nr_legacy_irqs() ? ~PIC_IRQS : ~0UL; > >>>> + > >>>> + /* Set up IO-APIC IRQ routing. */ > >>>> + x86_init.mpparse.setup_ioapic_ids(); > >>>> + sync_Arb_IDs(); > >>>> + } > Hi Konrad, > Enabling above code for Xen dom0 will cause following warning > because it writes a special value to ICR register. > [ 3.394981] ------------[ cut here ]------------ > [ 3.394985] WARNING: CPU: 0 PID: 1 at arch/x86/xen/enlighten.c:968 > xen_apic_write+0x15/0x20() > [ 3.394988] Modules linked in: > [ 3.394991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.19.0-rc3+ #5 > [ 3.394993] Hardware name: Dell Inc. OptiPlex 9020/0DNKMN, BIOS A03 > 09/17/2013 > [ 3.394996] 00000000000003c8 ffff88003056bdd8 ffffffff817611bb > 00000000000003c8 > [ 3.395000] 0000000000000000 ffff88003056be18 ffffffff8106f4ea > 0000000000000008 > [ 3.395004] ffffffff81fc1120 ffff880030561348 000000000000a108 > 000000000000a101 > [ 3.395008] Call Trace: > [ 3.395012] [] dump_stack+0x4f/0x6c > [ 3.395015] [] warn_slowpath_common+0xaa/0xd0 > [ 3.395018] [] warn_slowpath_null+0x15/0x20 > [ 3.395021] [] xen_apic_write+0x15/0x20 > [ 3.395026] [] sync_Arb_IDs+0x84/0x86 > [ 3.395029] [] setup_IO_APIC+0x7f/0x8e3 > [ 3.395033] [] ? trace_hardirqs_on+0xd/0x10 > [ 3.395036] [] ? _raw_spin_unlock_irqrestore+0x8a/0xa0 > [ 3.395040] [] xen_smp_prepare_cpus+0x5d/0x184 > [ 3.395044] [] kernel_init_freeable+0x149/0x293 > [ 3.395047] [] ? kernel_init+0x9/0xf0 > [ 3.395049] [] ? rest_init+0xd0/0xd0 > [ 3.395052] [] kernel_init+0x9/0xf0 > [ 3.395054] [] ret_from_fork+0x7c/0xb0 > [ 3.395057] [] ? rest_init+0xd0/0xd0 > [ 3.395066] ---[ end trace 7c4371c8ba33d5d0 ]--- > > > >>>> ioapic_initialized = 1; > >>>> + > >>>> + if (!xen_smp) { > >>>> + init_IO_APIC_traps(); > >>>> + if (nr_legacy_irqs()) > >>>> + check_timer(); > >>>> + } > >>>> } > And enabling above code causes Xen dom0 reboots. Which is due to the 'check_timer' trying to setup its timer and failing and then moving under its feet the legacy_pic to the NULL one and then hitting panic. The 'check_timer' has the logic to swap the 'legacy_pic': 2186 legacy_pic->init(1); which ends up executing: 317 new_val = inb(PIC_MASTER_IMR); 318 if (new_val != probe_val) { 319 printk(KERN_INFO "Using NULL legacy PIC\n"); 320 legacy_pic = &null_legacy_pic; 321 raw_spin_unlock_irqrestore(&i8259A_lock, flags); 322 return; 323 } And the 'legacy_pic' has now be swapped over to the 'null_legacy_pic' for which: 2393 if (nr_legacy_irqs()) 2394 check_timer(); 2395 70 static inline int nr_legacy_irqs(void) 71 { 72 return legacy_pic->nr_legacy_irqs; 73 } 74 would return zero (and not invoke the 'check_timer'), but because we do make the check inside the 'check_timer' we continue on. Perhaps something like this? The patch of course ignores the WARN which woudl need something else. > Haven't test HVM and PV kernel yet. > So seems we still need special treatment for xen here. > Regards! > Gerry > > >>>> > >>>> /* > >>>> diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c > >>>> index 4c071aeb8417..7eb0283901fa 100644 > >>>> --- a/arch/x86/xen/smp.c > >>>> +++ b/arch/x86/xen/smp.c > >>>> @@ -326,7 +326,10 @@ static void __init xen_smp_prepare_cpus(unsigned int max_cpus) > >>>> > >>>> xen_raw_printk(m); > >>>> panic(m); > >>>> + } else { > >>>> + setup_IO_APIC(true); > >>>> } > >>>> + > >>>> xen_init_lock_cpu(0); > >>>> > >>>> smp_store_boot_cpu_info(); > >>>> -- > >>>> 1.7.10.4 > >>>> > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > >>> the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > >>> Please read the FAQ at http://www.tux.org/lkml/ > >>> > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c index 3f5f604..e474389 100644 --- a/arch/x86/kernel/apic/io_apic.c +++ b/arch/x86/kernel/apic/io_apic.c @@ -2184,6 +2184,14 @@ static inline void __init check_timer(void) */ apic_write(APIC_LVT0, APIC_LVT_MASKED | APIC_DM_EXTINT); legacy_pic->init(1); + /* + * The init swapped out the legacy_pic to point to the NULL one. + * As such we should not even have entered this init routine + * (which depends on ->nr_legacy_irqs having an non-zero value + * and null_legacy_pic has zero. + */ + if (legacy_pic == null_legacy_pic) + goto out; pin1 = find_isa_irq_pin(0, mp_INT); apic1 = find_isa_irq_apic(0, mp_INT); diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c index 4c071ae..9f404df 100644 --- a/arch/x86/xen/smp.c +++ b/arch/x86/xen/smp.c @@ -327,6 +327,7 @@ static void __init xen_smp_prepare_cpus(unsigned int max_cpus) xen_raw_printk(m); panic(m); } + setup_IO_APIC(); xen_init_lock_cpu(0); smp_store_boot_cpu_info();