From patchwork Fri Feb 17 18:20:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tamas K Lengyel X-Patchwork-Id: 9580503 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B4C8C6043A for ; Fri, 17 Feb 2017 18:23:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5D642867D for ; Fri, 17 Feb 2017 18:23:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 99CB8285E7; Fri, 17 Feb 2017 18:23:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 119E4285E7 for ; Fri, 17 Feb 2017 18:23:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cen9n-00065n-7r; Fri, 17 Feb 2017 18:21:23 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cen9l-00065h-F7 for xen-devel@lists.xen.org; Fri, 17 Feb 2017 18:21:21 +0000 Received: from [85.158.139.211] by server-5.bemta-5.messagelabs.com id F5/38-02139-02F37A85; Fri, 17 Feb 2017 18:21:20 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrIIsWRWlGSWpSXmKPExsXiVRvkoitvvzz CYNstM4slHxezODB6HN39mymAMYo1My8pvyKBNWPvlUusBf+0Kp7N28TSwHhbuYuRk0NIYAKj RMMTIJuLg0VgFqvEihffWEAcCYFtrBJHl8xn72LkAHJyJLpX1UCYJRLr71eA9PIKCEqcnPmEB WKOl8TL84cZQWxOAXuJ4+1LmUHGCAl8YZbovraPCSTBJmAo8WjPV2YQm0VAVWLVx4NsIDN5BQ IkmpvFQMLCAjYS7YsPs4LYIgLKEr2/foPNZxYol1jVMYcZpJxZQF1i/TwhiLC2xLKFr5knMAr OQnLRLISqWUiqFjAyr2LUKE4tKkst0jUy1UsqykzPKMlNzMzRNTQw1ctNLS5OTE/NSUwq1kvO z93ECAzYegYGxh2Mu9r9DjFKcjApifI+4loeIcSXlJ9SmZFYnBFfVJqTWnyIUYaDQ0mC18MOK CdYlJqeWpGWmQOMHZi0BAePkghvJUiat7ggMbc4Mx0idYrRmOPLzjMvmTh27br8kkmIJS8/L1 VKnJcHpFQApDSjNA9uECymLzHKSgnzMjIwMAjxFKQW5WaWoMq/YhTnYFQS5rUBmcKTmVcCt+8 V0ClMQKd0RiwFOaUkESEl1cDIwrtX+NXiR4/lbx547VnQrvLs8YWb1xPOPV6/k8OuZ+7PjONL 68sS5876mDkhoun3tqO3K3SuL7wvXDWD85OCbkFVesx23oid23Rb1qrKNsasnrl6eemky1Llu +qFHrzgeHKN8+bZHT7X/p0vnfl/y+PH27gmfZNLDZfbVssxbd2JKaVdnO7l25VYijMSDbWYi4 oTAR25YGjkAgAA X-Env-Sender: tamas.k.lengyel@gmail.com X-Msg-Ref: server-10.tower-206.messagelabs.com!1487355679!68333797!1 X-Originating-IP: [74.125.82.68] X-SpamReason: No, hits=0.8 required=7.0 tests=BODY_RANDOM_LONG, RCVD_BY_IP X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 35402 invoked from network); 17 Feb 2017 18:21:19 -0000 Received: from mail-wm0-f68.google.com (HELO mail-wm0-f68.google.com) (74.125.82.68) by server-10.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 17 Feb 2017 18:21:19 -0000 Received: by mail-wm0-f68.google.com with SMTP id c85so3921689wmi.1 for ; Fri, 17 Feb 2017 10:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=J2IEbsZqpRRTZYPj1rbLMUrh4iYNYlDXXh0nLO/VgWk=; b=g1iPlmFd5rRpLBbkq/yoSy21k8/V8FEyKHwqfCTDJGTUmhXRkGHtlS2kG1HYmJCXpa 3ngEbj0J7eSO4v4bko3m28y5umAffbJI7YLw7jCvwaGAPdO4boZLJ1lWZrXPlQzLP09a EKQlM1JwGbjttAH+iKbD28yT7EEhrLgaCbpW+4Y8xHpmaJllVFCmhjB5CLSW3FDG8FfD lZ5NuTCh9mAUKNKUi0fIUtsxUPaAU7hDZmeXaKy1QiXjbL/s7rMao5H/hTXHJ/aDFE3T bHpUiPlBVbsSCE60CVGdo4SkDB05slEIX1PrEVMMJwpw9lgHopQVNRsya8IvQE6tWhSl 4Gaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=J2IEbsZqpRRTZYPj1rbLMUrh4iYNYlDXXh0nLO/VgWk=; b=K1n445O5wERQqUX3FZSakDuXdMijH1fgGs6QPisGNXUP1JVK4CiufxR99b7Big0tz2 gX4yn40F6ZcEabZMdOPCi5Ri0I9PuX2YTB4AXg38/HDsYQcXRJSYaPlgXZlamPOx84lR UM+yf53pta+mA7Vz3PPOJzDIZdNN77FPku+JO7mVBDzKbI1pTmMA8T3Ga9iFA7mi9jEE h+kjstexEoxQ8eVyBmI12eQ1PQpXL0/tM63PmEfPhhkX2vltSKn4YA2CdrnnT9ESle4c 3KJF2hWbVXss67ZQabOC2qCwU+7dv/eF7T+SYf5R6FAcR+XVElYtZOm671O7EkSwaare sUCg== X-Gm-Message-State: AMke39kUkEZv3j3E8P8PqcmNF5eXLAJ85co5Zd98bfc1osTmNjEsB4wSPGH080HnM84/lQX8hWVTqcWXaobFKA== X-Received: by 10.28.214.144 with SMTP id n138mr4867897wmg.136.1487355679375; Fri, 17 Feb 2017 10:21:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.73.212 with HTTP; Fri, 17 Feb 2017 10:20:38 -0800 (PST) In-Reply-To: <58A6C079020000780013B1A6@prv-mh.provo.novell.com> References: <836159e1-2705-05a7-689f-95d783c2e57b@citrix.com> <58A43595020000780013A0F2@prv-mh.provo.novell.com> <58A57F92020000780013A929@prv-mh.provo.novell.com> <58A6C079020000780013B1A6@prv-mh.provo.novell.com> From: Tamas K Lengyel Date: Fri, 17 Feb 2017 11:20:38 -0700 Message-ID: To: Jan Beulich Cc: Andrew Cooper , Kevin Tian , Xen-devel Subject: Re: [Xen-devel] Unable to boot Xen 4.8 with iommu=0 X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Feb 17, 2017 at 1:20 AM, Jan Beulich wrote: >>>> On 17.02.17 at 07:53, wrote: >>> From: Tian, Kevin >>> Sent: Friday, February 17, 2017 11:35 AM >>> > >> >>> > >> Or wait - do you have the same issue if you use >>> > >> "iommu=no,no-intremap"? In which case the problem would be >>> > >> that "iommu=no" should clear more than just "iommu_enable", or >>> > >> code checking iommu_intremap early (before iommu_setup() >>> > >> manages to clear it in the case here) would need to made look at >>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if >>> > >> both variables are clear, which suggests to me that >>> > >> iommu_enable is intended to have two different meanings in >>> > >> different contexts (master flag vs. controlling just DMA >>> > >> remapping). Kevin, Feng - any thoughts here? >>> > > >>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default" >>> > >>> > Thanks for confirming. >>> > >>> > Kevin, Feng, we now depend on your input regarding the intentions >>> > with the two variables. >>> > >>> >>> Feng just left Intel. Let me take a look at code to understand the >>> rationale behind. >>> >> >> Jan, looks it's caused by your change back to 2012: >> >> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f >> Author: Jan Beulich >> Date: Fri Nov 2 17:15:30 2012 +0100 > > Oh, I see. That's been a while; I'm sorry for not having remembered > and bugging you. > >> Before that iommu_enable was the master flag consistently. I'm still >> trying to understand the background and you may help elaborate if >> still something in your memory. > > I think the commit description is pretty clear. Especially this part > > "This could have the nice side effect of allowing to use "iommu=off" > even when x2APIC was pre-enabled by the BIOS (in which case interrupt > remapping is a requirement, but DMA translation [obviously] isn't), but > that doesn't currently work (and hence x2apic_bsp_setup() forces the > IOMMU on rather than just interrupt remapping)." > > is quite relevant in the context here. Just like Linux, we really ought > to have a way to run with interrupt remapping, but without DMA > remapping. So I think this old commit was a half-hearted step into > that direction, not recognizing that it would break some other case. > > So I think the only reasonable way forward to address Tamas's > issue is to fully disentangle DMA and interrupt remapping setup, > which I'm sure is going to take some time. As a minimal adjustment, > though, I wonder whether that old commit's adjustment to apic.c > shouldn't rather have set force_intremap. Tamas - could you check > whether > > --- a/xen/arch/x86/apic.c > +++ b/xen/arch/x86/apic.c > @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void) > goto restore_out; > } > > - force_iommu = 1; > + force_intremap = 1; > > genapic = apic_x2apic_probe(); > printk("Switched to APIC driver %s.\n", genapic->name); > > makes things any better? > > Jan Just making this change won't compile: apic.c: In function ‘x2apic_bsp_setup’: apic.c:947:5: error: ‘force_intremap’ undeclared (first use in this function) force_intremap = 1; ^ Had to also add With these changes in place the system starts to boot but dom0 fails to assemble the mdraid disk and find root: Begin: Running /scripts/local-block ... mdadm: No devices listed in conf file were found. WARNING: Failed to connect to lvmetad. Falling back to device scanning. Volume group "vg" not found Cannot process volume group vg done. done. Gave up waiting for root device. Common problems: - Boot args (cat /proc/cmdline) - Check rootdelay= (did the system wait long enough?) - Check root= (did the system wait for the right device?) - Missing modules (cat /proc/modules; ls /dev) ALERT! /dev/mapper/vg-root does not exist. Dropping to a shell! Also, I see Xen still using the x2apic_cluster: (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Switched to APIC driver x2apic_cluster. With iommu=1 everything works as normal. Tamas diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c index d793f5de1a..a45578608b 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -52,6 +52,7 @@ custom_param("iommu", parse_iommu_param); bool_t __initdata iommu_enable = 1; bool_t __read_mostly iommu_enabled; bool_t __read_mostly force_iommu; +bool_t __read_mostly force_intremap; bool_t __hwdom_initdata iommu_dom0_strict; bool_t __read_mostly iommu_verbose; bool_t __read_mostly iommu_workaround_bios_bug; @@ -364,7 +365,7 @@ int iommu_iotlb_flush_all(struct domain *d) int __init iommu_setup(void) { int rc = -ENODEV; - bool_t force_intremap = force_iommu && iommu_intremap; + force_intremap = force_iommu && iommu_intremap; if ( iommu_dom0_strict ) iommu_passthrough = 0; diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 5803e3f95b..770e4f6282 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -29,6 +29,7 @@ extern bool_t iommu_enable, iommu_enabled; extern bool_t force_iommu, iommu_verbose; +extern bool_t force_intremap; extern bool_t iommu_workaround_bios_bug, iommu_igfx, iommu_passthrough; extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost; extern bool_t iommu_hap_pt_share;