[13/16] kvm: enable MSI-X capabilty for assigned device

Commit Message

Sheng Yang April 8, 2009, 2:26 a.m. UTC

On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote:
> On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote:
> > On Saturday 04 April 2009 05:27:43 Alex Williamson wrote:
> > > Do we need some disable logic here?  If I toggle a bnx2 NIC in a guest,
> > > I get the following when it attempts to come back up:
> > >
> > > MSI-X entry number is zero!
> > > assigned_dev_update_msix_mmio: No such device or address
> >
> > It seems that driver didn't fill the MMIO with any correct MSIX
> > information, or the program fail to intercept it after driver set enable
> > bit of MSIX. It's strange... (Have it got something to do with PM and
> > some EXP feature you mentioned?)
>
> My guess was that it filled in the MSIX info, but then can't find a free
> slot to reload the MSIX data when it tries to re-enable MSIX.  I hacked
> the bnx2 driver to not rely on PM and EXP capabilities for this test, it
> seems to work, but it's possible that I broke something.  My host also
> locks up the second time I try to export this device to a guest, maybe a
> problem with my bnx2 hacks, MSIX not getting reset, or prototype
> hardware.  I'll see if I can find another MSIX capable device to export
> to a guest.
>
> > Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in
> > qemu/hw/device-assignment.c and post the output?
>
> Yup, see below.  The error comes after I 'ifdown eth0; ifup eth0' in the
> guest.  Note bnx2 appears to only turn on MSIX for SMP systems.  Thanks,
>
> Alex

Seems your "ifdown/ifup" script reload the module? Oh god, I found one bug 
after checked the spec:

System software reads this field to determine the MSI-X Table Size *N*, which 
is encoded as *N-1*. For example, a returned value of “00000000011” indicates 
a table size of 4.

But it seems still can't explain the problem...(OK, it may affect the guest in 
a unknown way as well...) I would post a fix for it soon.

> val=0x00000008 len=2 assigned_dev_pci_write_config: (4.0): address=0052
> val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0006
> val=0x00000010 len=2 assigned_dev_pci_read_config: (4.0): address=0034
> val=0x00000040 len=1 assigned_dev_pci_read_config: (4.0): address=0040
> val=0x00000005 len=1 assigned_dev_pci_read_config: (4.0): address=0041
> val=0x00000050 len=1 assigned_dev_pci_read_config: (4.0): address=0050
> val=0x00000011 len=1 assigned_dev_pci_read_config: (4.0): address=0052
> val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0054
> val=0x0000c000 len=4 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x0, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x4, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x8, val 0x4191 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x10, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x14, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x18, val 0x4199 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x20, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x24, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x28, val 0x41a1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x30, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x34, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x38, val 0x41a9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x40, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x44, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x48, val 0x41b1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x50, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x54, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x58, val 0x41b9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x60, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x64, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x68, val 0x41c1 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x70, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x74, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x78, val 0x41c9 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x80, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table
> mmio offset 0x84, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio
> offset 0x88, val 0x41d1 assigned_dev_pci_read_config: (4.0): address=0004
> val=0x00000046 len=2 assigned_dev_pci_write_config: (4.0): address=0004
> val=0x00000446 len=2 assigned_dev_pci_write_config: NON BAR (4.0):

The writing to MMIO have been intercepted, but code fail to count it? 
Strange...

Could you try this debug?

         memcpy(&msg_ctrl, va + i * 16 + 12, 4);

(Further more, you can check the content of msg_ctrl when code try to count 
them). And also please try the fix for MSI-X table size.

Thanks!

Comments

Alex Williamson April 8, 2009, 4:13 p.m. UTC | #1

Hi Sheng,

On Wed, 2009-04-08 at 10:26 +0800, Sheng Yang wrote:
> On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote:
> > On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote:
> > > Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in
> > > qemu/hw/device-assignment.c and post the output?
> >
> > Yup, see below.  The error comes after I 'ifdown eth0; ifup eth0' in the
> > guest.  Note bnx2 appears to only turn on MSIX for SMP systems.  Thanks,
> >
> > Alex
> 
> Seems your "ifdown/ifup" script reload the module?

No, the bnx2 module isn't unloaded on ifdown.

>  Oh god, I found one bug 
> after checked the spec:
> 
> System software reads this field to determine the MSI-X Table Size *N*, which 
> is encoded as *N-1*. For example, a returned value of “00000000011” indicates 
> a table size of 4.
> 
> But it seems still can't explain the problem...(OK, it may affect the guest in 
> a unknown way as well...) I would post a fix for it soon.
[snip]
> 
> The writing to MMIO have been intercepted, but code fail to count it? 
> Strange...
> 
> Could you try this debug?

I added the debug printfs, plus the MSI-X table size patch, and printed
the value of msg_ctrl as we loop through.  Output below.  This is what
made me think the MSI-X state isn't getting cleared when the driver
closes the interface.  Let me know what you think.  Thanks,

Alex

init_assigned_device: Registering real physical device 03:00.0 (bus=3 dev=0 func=0)
get_real_device: region 0 size 33554432 start 0xf4000000 type 512 resource_fd 19
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000014e4 len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00001639 len=2
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000014e4 len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00001639 len=2
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000014e4 len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00001639 len=2
assigned_dev_pci_read_config: (4.0): address=000a val=0x00000200 len=2
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000014e4 len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00001639 len=2
assigned_dev_pci_write_config: (4.0): address=0010 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe000000 len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0xf4000000 len=4
assigned_dev_iomem_map: e_phys=f4000000 r_virt=0x7f4fd9bfc000 type=0 len=02000000 region_num=0 
assigned_dev_iomem_map: munmap done, virt_base 0x0x7f4fd9c08000
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: (4.0): address=0014 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0018 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0020 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0xffffffff len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0xffff0001 len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0xffff0001 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_read_config: (4.0): address=003d val=0x00000001 len=1
assigned_dev_pci_write_config: (4.0): address=003c val=0x0000000b len=1
assigned_dev_pci_read_config: (4.0): address=000a val=0x00000200 len=2
assigned_dev_pci_read_config: (4.0): address=0000 val=0x000014e4 len=2
assigned_dev_pci_read_config: (4.0): address=0002 val=0x00001639 len=2
assigned_dev_pci_read_config: (4.0): address=000e val=0x00000080 len=1
assigned_dev_pci_read_config: (4.0): address=0000 val=0x163914e4 len=4
assigned_dev_pci_read_config: (4.0): address=000e val=0x00000080 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0008 val=0x02000020 len=4
assigned_dev_pci_read_config: (4.0): address=003d val=0x00000001 len=1
assigned_dev_pci_read_config: (4.0): address=003c val=0x0000000b len=1
assigned_dev_pci_read_config: (4.0): address=0010 val=0xf4000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0xfe000000 len=4
assigned_dev_pci_write_config: (4.0): address=0010 val=0xf4000000 len=4
assigned_dev_iomem_map: e_phys=f4000000 r_virt=0x7f4fd9bfc000 type=0 len=02000000 region_num=0 
BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1)
assigned_dev_iomem_map: munmap done, virt_base 0x0x7f4fd9c08000
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0014 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0018 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0020 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0xffffffff len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000000 len=4
assigned_dev_pci_write_config: (4.0): address=0024 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0xfffffffe len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0xfffffffe len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0xffff0000 len=4
assigned_dev_pci_write_config: (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_read_config: (4.0): address=002c val=0x0000103c len=2
assigned_dev_pci_read_config: (4.0): address=002e val=0x00007055 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000442 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_read_config: (4.0): address=000d val=0x00000000 len=1
assigned_dev_pci_write_config: (4.0): address=000d val=0x00000040 len=1
assigned_dev_pci_write_config: NON BAR (4.0): address=000d val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0000 val=0x163914e4 len=4
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00100446 len=4
assigned_dev_pci_read_config: (4.0): address=0008 val=0x02000020 len=4
assigned_dev_pci_read_config: (4.0): address=000c val=0x00800010 len=4
assigned_dev_pci_read_config: (4.0): address=0010 val=0xf4000000 len=4
assigned_dev_pci_read_config: (4.0): address=0014 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0018 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=001c val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0020 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0024 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=0028 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=002c val=0x7055103c len=4
assigned_dev_pci_read_config: (4.0): address=0030 val=0x00010000 len=4
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=4
assigned_dev_pci_read_config: (4.0): address=0038 val=0x00000000 len=4
assigned_dev_pci_read_config: (4.0): address=003c val=0x0000010b len=4
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_write_config: (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0051 val=0x00000000 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_write_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0054 val=0x0000c000 len=4
msix_mmio_writel: write to MSI-X entry table mmio offset 0x0, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x8, val 0x4149
msix_mmio_writel: write to MSI-X entry table mmio offset 0x10, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x14, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x18, val 0x4151
msix_mmio_writel: write to MSI-X entry table mmio offset 0x20, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x24, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x28, val 0x4159
msix_mmio_writel: write to MSI-X entry table mmio offset 0x30, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x34, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x38, val 0x4161
msix_mmio_writel: write to MSI-X entry table mmio offset 0x40, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x44, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x48, val 0x4169
msix_mmio_writel: write to MSI-X entry table mmio offset 0x50, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x54, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x58, val 0x4171
msix_mmio_writel: write to MSI-X entry table mmio offset 0x60, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x64, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x68, val 0x4179
msix_mmio_writel: write to MSI-X entry table mmio offset 0x70, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x74, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x78, val 0x4181
msix_mmio_writel: write to MSI-X entry table mmio offset 0x80, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x84, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x88, val 0x4189
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000046 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_write_config: (4.0): address=0052 val=0x00008008 len=2
the MSIX capabilty position is 0x50
the MSIX entries_max_nr is 0x9
0: msg_ctrl: 0000
1: msg_ctrl: 0000
2: msg_ctrl: 0000
3: msg_ctrl: 0000
4: msg_ctrl: 0000
5: msg_ctrl: 0000
6: msg_ctrl: 0000
7: msg_ctrl: 0000
8: msg_ctrl: 0000
assigned_dev_update_msix_mmio: MSI-X data 0x4149, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x18, entry 0
!assigned_dev_update_msix_mmio: MSI-X data 0x4151, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x19, entry 1
!assigned_dev_update_msix_mmio: MSI-X data 0x4159, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1a, entry 2
!assigned_dev_update_msix_mmio: MSI-X data 0x4161, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1b, entry 3
!assigned_dev_update_msix_mmio: MSI-X data 0x4169, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1c, entry 4
!assigned_dev_update_msix_mmio: MSI-X data 0x4171, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1d, entry 5
!assigned_dev_update_msix_mmio: MSI-X data 0x4179, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1e, entry 6
!assigned_dev_update_msix_mmio: MSI-X data 0x4181, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x1f, entry 7
!assigned_dev_update_msix_mmio: MSI-X data 0x4189, MSI-X addr_lo 0xfeeff00c
!assigned_dev_update_msix_mmio: MSI-X entry gsi 0x20, entry 8
!msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x0
assigned_dev_pci_write_config: (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_write_config: (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0068 val=0x00000088 len=4
msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00008008 len=2
assigned_dev_pci_write_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000046 len=2
msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x8c, val 0x1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_write_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_read_config: (4.0): address=0054 val=0x0000c000 len=4
msix_mmio_writel: write to MSI-X entry table mmio offset 0x0, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x8, val 0x4191
msix_mmio_writel: write to MSI-X entry table mmio offset 0x10, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x14, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x18, val 0x4199
msix_mmio_writel: write to MSI-X entry table mmio offset 0x20, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x24, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x28, val 0x41a1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x30, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x34, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x38, val 0x41a9
msix_mmio_writel: write to MSI-X entry table mmio offset 0x40, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x44, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x48, val 0x41b1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x50, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x54, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x58, val 0x41b9
msix_mmio_writel: write to MSI-X entry table mmio offset 0x60, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x64, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x68, val 0x41c1
msix_mmio_writel: write to MSI-X entry table mmio offset 0x70, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x74, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x78, val 0x41c9
msix_mmio_writel: write to MSI-X entry table mmio offset 0x80, val 0xfeeff00c
msix_mmio_writel: write to MSI-X entry table mmio offset 0x84, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x88, val 0x41d1
assigned_dev_pci_read_config: (4.0): address=0004 val=0x00000046 len=2
assigned_dev_pci_write_config: (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_write_config: NON BAR (4.0): address=0004 val=0x00000446 len=2
assigned_dev_pci_read_config: (4.0): address=0006 val=0x00000010 len=2
assigned_dev_pci_read_config: (4.0): address=0034 val=0x00000040 len=1
assigned_dev_pci_read_config: (4.0): address=0040 val=0x00000005 len=1
assigned_dev_pci_read_config: (4.0): address=0041 val=0x00000050 len=1
assigned_dev_pci_read_config: (4.0): address=0050 val=0x00000011 len=1
assigned_dev_pci_read_config: (4.0): address=0052 val=0x00000008 len=2
assigned_dev_pci_write_config: (4.0): address=0052 val=0x00008008 len=2
the MSIX capabilty position is 0x50
the MSIX entries_max_nr is 0x9
0: msg_ctrl: 0001
1: msg_ctrl: 0001
2: msg_ctrl: 0001
3: msg_ctrl: 0001
4: msg_ctrl: 0001
5: msg_ctrl: 0001
6: msg_ctrl: 0001
7: msg_ctrl: 0001
8: msg_ctrl: 0001
MSI-X entry number is zero!
assigned_dev_update_msix_mmio: No such device or address
msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x0
msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x0
assigned_dev_pci_write_config: (4.0): address=0068 val=0x00000088 len=4
assigned_dev_pci_write_config: NON BAR (4.0): address=0068 val=0x00000088 len=4


--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Sheng Yang April 9, 2009, 1:31 a.m. UTC | #2

On Thursday 09 April 2009 00:13:56 Alex Williamson wrote:
> Hi Sheng,
>
> On Wed, 2009-04-08 at 10:26 +0800, Sheng Yang wrote:
> > On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote:
> > > On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote:
> > > > Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in
> > > > qemu/hw/device-assignment.c and post the output?
> > >
> > > Yup, see below.  The error comes after I 'ifdown eth0; ifup eth0' in
> > > the guest.  Note bnx2 appears to only turn on MSIX for SMP systems. 
> > > Thanks,
> > >
> > > Alex
> >
> > Seems your "ifdown/ifup" script reload the module?
>
> No, the bnx2 module isn't unloaded on ifdown.
>
> >  Oh god, I found one bug
> > after checked the spec:
> >
> > System software reads this field to determine the MSI-X Table Size *N*,
> > which is encoded as *N-1*. For example, a returned value of “00000000011”
> > indicates a table size of 4.
> >
> > But it seems still can't explain the problem...(OK, it may affect the
> > guest in a unknown way as well...) I would post a fix for it soon.
>
> [snip]
>
> > The writing to MMIO have been intercepted, but code fail to count it?
> > Strange...
> >
> > Could you try this debug?
>
> I added the debug printfs, plus the MSI-X table size patch, and printed
> the value of msg_ctrl as we loop through.  Output below.  This is what
> made me think the MSI-X state isn't getting cleared when the driver
> closes the interface.  Let me know what you think.  Thanks,
>

Thanks Alex, now I know where the problem is. Part of functional haven't been 
implemented...

> address=0052 val=0x00008008 len=2 the MSIX capabilty position is 0x50
> the MSIX entries_max_nr is 0x9
> 0: msg_ctrl: 0001
> 1: msg_ctrl: 0001
> 2: msg_ctrl: 0001
> 3: msg_ctrl: 0001
> 4: msg_ctrl: 0001
> 5: msg_ctrl: 0001
> 6: msg_ctrl: 0001
> 7: msg_ctrl: 0001
> 8: msg_ctrl: 0001
> MSI-X entry number is zero!
> assigned_dev_update_msix_mmio: No such device or address

Driver write to the vectors at first, then enable MSI-X,

> msix_mmio_writel: write to MSI-X entry table mmio offset 0xc, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x1c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x2c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x3c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x4c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x5c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x6c, val 0x0
> msix_mmio_writel: write to MSI-X entry table mmio offset 0x7c, val 0x0

And finally clear the mask bit.

For current we didn't implement mask capability in MSI-X vectors, so it won't 
work...

OK. I'd like to remove the check of mask bit and only ignored unused vector 
when msg data is zero now(hope it won't cause more problems). And we would add 
support for per-vector mask later.

Thanks for help to debug!

Patch

diff --git a/qemu/hw/device-assignment.c b/qemu/hw/device-assignment.c
index 09e54ae..ba31bed 100644
--- a/qemu/hw/device-assignment.c
+++ b/qemu/hw/device-assignment.c
@@ -45,7 +45,7 @@ 
 #define IORESOURCE_DMA      0x00000800
 #define IORESOURCE_PREFETCH 0x00001000  /* No side effects */

-/* #define DEVICE_ASSIGNMENT_DEBUG 1 */
+#define DEVICE_ASSIGNMENT_DEBUG 1

 #ifdef DEVICE_ASSIGNMENT_DEBUG
 #define DEBUG(fmt, ...)                                       \
@@ -816,9 +816,13 @@  static int assigned_dev_update_msix_mmio(PCIDevice 
*pci_dev)
     else
         pos = pci_dev->cap.start;

+    fprintf(stderr, "the MSIX capabilty position is 0x%lx\n", pos);
+
     entries_max_nr = pci_dev->config[pos + 2];
     entries_max_nr &= PCI_MSIX_TABSIZE;

+    fprintf(stderr, "the MSIX entries_max_nr is 0x%lx\n", entries_max_nr);
+
     /* Get the usable entry number for allocating */
     for (i = 0; i < entries_max_nr; i++) {