Unable to boot Xen 4.8 with iommu=0

Commit Message

Tamas K Lengyel Feb. 17, 2017, 6:20 p.m. UTC

On Fri, Feb 17, 2017 at 1:20 AM, Jan Beulich <[e-mail]> wrote:
>>>> On 17.02.17 at 07:53, <[e-mail]> wrote:
>>>  From: Tian, Kevin
>>> Sent: Friday, February 17, 2017 11:35 AM
>>> > >>
>>> > >> Or wait - do you have the same issue if you use
>>> > >> "iommu=no,no-intremap"? In which case the problem would be
>>> > >> that "iommu=no" should clear more than just "iommu_enable", or
>>> > >> code checking iommu_intremap early (before iommu_setup()
>>> > >> manages to clear it in the case here) would need to made look at
>>> > >> both variables. Oddly enough acpi_parse_dmar() only bails if
>>> > >> both variables are clear, which suggests to me that
>>> > >> iommu_enable is intended to have two different meanings in
>>> > >> different contexts (master flag vs. controlling just DMA
>>> > >> remapping). Kevin, Feng - any thoughts here?
>>> > >
>>> > > iommu=no,no-intremap boots fine with "(XEN) Using APIC driver default"
>>> >
>>> > Thanks for confirming.
>>> >
>>> > Kevin, Feng, we now depend on your input regarding the intentions
>>> > with the two variables.
>>> >
>>>
>>> Feng just left Intel. Let me take a look at code to understand the
>>> rationale behind.
>>>
>>
>> Jan, looks it's caused by your change back to 2012:
>>
>> commit 7a8f6d0607a38c64506b4e8b473d955bf8e2a71f
>> Author: Jan Beulich <[e-mail]>
>> Date:   Fri Nov 2 17:15:30 2012 +0100
>
> Oh, I see. That's been a while; I'm sorry for not having remembered
> and bugging you.
>
>> Before that iommu_enable was the master flag consistently. I'm still
>> trying to understand the background and you may help elaborate if
>> still something in your memory.
>
> I think the commit description is pretty clear. Especially this part
>
> "This could have the nice side effect of allowing to use "iommu=off"
>  even when x2APIC was pre-enabled by the BIOS (in which case interrupt
>  remapping is a requirement, but DMA translation [obviously] isn't), but
>  that doesn't currently work (and hence x2apic_bsp_setup() forces the
>  IOMMU on rather than just interrupt remapping)."
>
> is quite relevant in the context here. Just like Linux, we really ought
> to have a way to run with interrupt remapping, but without DMA
> remapping. So I think this old commit was a half-hearted step into
> that direction, not recognizing that it would break some other case.
>
> So I think the only reasonable way forward to address Tamas's
> issue is to fully disentangle DMA and interrupt remapping setup,
> which I'm sure is going to take some time. As a minimal adjustment,
> though, I wonder whether that old commit's adjustment to apic.c
> shouldn't rather have set force_intremap. Tamas - could you check
> whether
>
> --- a/xen/arch/x86/apic.c
> +++ b/xen/arch/x86/apic.c
> @@ -975,7 +975,7 @@ void __init x2apic_bsp_setup(void)
>          goto restore_out;
>      }
>
> -    force_iommu = 1;
> +    force_intremap = 1;
>
>      genapic = apic_x2apic_probe();
>      printk("Switched to APIC driver %s.\n", genapic->name);
>
> makes things any better?
>
> Jan

Just making this change won't compile:

apic.c: In function ‘x2apic_bsp_setup’:
apic.c:947:5: error: ‘force_intremap’ undeclared (first use in this function)
     force_intremap = 1;
     ^

Had to also add


With these changes in place the system starts to boot but dom0 fails
to assemble the mdraid disk and find root:

Begin: Running /scripts/local-block ... mdadm: No devices listed in
conf file were found.
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
  Volume group "vg" not found
  Cannot process volume group vg
done.
done.
Gave up waiting for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/vg-root does not exist.  Dropping to a shell!

Also, I see Xen still using the x2apic_cluster:

(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.

With iommu=1 everything works as normal.

Tamas

Patch

diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iommu.c
index d793f5de1a..a45578608b 100644
--- a/xen/drivers/passthrough/iommu.c
+++ b/xen/drivers/passthrough/iommu.c
@@ -52,6 +52,7 @@  custom_param("iommu", parse_iommu_param);
 bool_t __initdata iommu_enable = 1;
 bool_t __read_mostly iommu_enabled;
 bool_t __read_mostly force_iommu;
+bool_t __read_mostly force_intremap;
 bool_t __hwdom_initdata iommu_dom0_strict;
 bool_t __read_mostly iommu_verbose;
 bool_t __read_mostly iommu_workaround_bios_bug;
@@ -364,7 +365,7 @@  int iommu_iotlb_flush_all(struct domain *d)
 int __init iommu_setup(void)
 {
     int rc = -ENODEV;
-    bool_t force_intremap = force_iommu && iommu_intremap;
+    force_intremap = force_iommu && iommu_intremap;

     if ( iommu_dom0_strict )
         iommu_passthrough = 0;
diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h
index 5803e3f95b..770e4f6282 100644
--- a/xen/include/xen/iommu.h
+++ b/xen/include/xen/iommu.h
@@ -29,6 +29,7 @@ 

 extern bool_t iommu_enable, iommu_enabled;
 extern bool_t force_iommu, iommu_verbose;
+extern bool_t force_intremap;
 extern bool_t iommu_workaround_bios_bug, iommu_igfx, iommu_passthrough;
 extern bool_t iommu_snoop, iommu_qinval, iommu_intremap, iommu_intpost;
 extern bool_t iommu_hap_pt_share;