From patchwork Wed Apr 8 02:26:01 2009 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sheng Yang X-Patchwork-Id: 16992 Received: from vger.kernel.org (vger.kernel.org [209.132.176.167]) by demeter.kernel.org (8.14.2/8.14.2) with ESMTP id n382R03i028755 for ; Wed, 8 Apr 2009 02:27:00 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758821AbZDHC0I (ORCPT ); Tue, 7 Apr 2009 22:26:08 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1755090AbZDHC0H (ORCPT ); Tue, 7 Apr 2009 22:26:07 -0400 Received: from mga10.intel.com ([192.55.52.92]:44387 "EHLO fmsmga102.fm.intel.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752120AbZDHC0F convert rfc822-to-8bit (ORCPT ); Tue, 7 Apr 2009 22:26:05 -0400 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP; 07 Apr 2009 19:22:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.39,341,1235980800"; d="scan'208";a="679703840" Received: from syang10-desktop.sh.intel.com (HELO syang10-desktop.localnet) ([10.239.13.189]) by fmsmga001.fm.intel.com with ESMTP; 07 Apr 2009 19:29:46 -0700 From: Sheng Yang Organization: Intel Opensource Technology Center To: kvm@vger.kernel.org Subject: Re: [kvm] [PATCH 13/16] kvm: enable MSI-X capabilty for assigned device Date: Wed, 8 Apr 2009 10:26:01 +0800 User-Agent: KMail/1.11.0 (Linux/2.6.27-11-generic; KDE/4.2.0; x86_64; ; ) Cc: Alex Williamson , Avi Kivity , Marcelo Tosatti , Anthony Liguori References: <1237261817-27955-1-git-send-email-sheng@linux.intel.com> <200904071409.10361.sheng@linux.intel.com> <1239122290.6184.86.camel@bling> In-Reply-To: <1239122290.6184.86.camel@bling> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200904081026.02816.sheng@linux.intel.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wednesday 08 April 2009 00:38:10 Alex Williamson wrote: > On Tue, 2009-04-07 at 14:09 +0800, Sheng Yang wrote: > > On Saturday 04 April 2009 05:27:43 Alex Williamson wrote: > > > Do we need some disable logic here? If I toggle a bnx2 NIC in a guest, > > > I get the following when it attempts to come back up: > > > > > > MSI-X entry number is zero! > > > assigned_dev_update_msix_mmio: No such device or address > > > > It seems that driver didn't fill the MMIO with any correct MSIX > > information, or the program fail to intercept it after driver set enable > > bit of MSIX. It's strange... (Have it got something to do with PM and > > some EXP feature you mentioned?) > > My guess was that it filled in the MSIX info, but then can't find a free > slot to reload the MSIX data when it tries to re-enable MSIX. I hacked > the bnx2 driver to not rely on PM and EXP capabilities for this test, it > seems to work, but it's possible that I broke something. My host also > locks up the second time I try to export this device to a guest, maybe a > problem with my bnx2 hacks, MSIX not getting reset, or prototype > hardware. I'll see if I can find another MSIX capable device to export > to a guest. > > > Could you enable DEVICE_ASSSIGNMENT_DEBUG=1 in > > qemu/hw/device-assignment.c and post the output? > > Yup, see below. The error comes after I 'ifdown eth0; ifup eth0' in the > guest. Note bnx2 appears to only turn on MSIX for SMP systems. Thanks, > > Alex Seems your "ifdown/ifup" script reload the module? Oh god, I found one bug after checked the spec: System software reads this field to determine the MSI-X Table Size *N*, which is encoded as *N-1*. For example, a returned value of “00000000011” indicates a table size of 4. But it seems still can't explain the problem...(OK, it may affect the guest in a unknown way as well...) I would post a fix for it soon. > val=0x00000008 len=2 assigned_dev_pci_write_config: (4.0): address=0052 > val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0006 > val=0x00000010 len=2 assigned_dev_pci_read_config: (4.0): address=0034 > val=0x00000040 len=1 assigned_dev_pci_read_config: (4.0): address=0040 > val=0x00000005 len=1 assigned_dev_pci_read_config: (4.0): address=0041 > val=0x00000050 len=1 assigned_dev_pci_read_config: (4.0): address=0050 > val=0x00000011 len=1 assigned_dev_pci_read_config: (4.0): address=0052 > val=0x00000008 len=2 assigned_dev_pci_read_config: (4.0): address=0054 > val=0x0000c000 len=4 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x0, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x4, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x8, val 0x4191 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x10, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x14, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x18, val 0x4199 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x20, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x24, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x28, val 0x41a1 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x30, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x34, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x38, val 0x41a9 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x40, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x44, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x48, val 0x41b1 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x50, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x54, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x58, val 0x41b9 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x60, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x64, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x68, val 0x41c1 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x70, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x74, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x78, val 0x41c9 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x80, val 0xfeeff00c msix_mmio_writel: write to MSI-X entry table > mmio offset 0x84, val 0x0 msix_mmio_writel: write to MSI-X entry table mmio > offset 0x88, val 0x41d1 assigned_dev_pci_read_config: (4.0): address=0004 > val=0x00000046 len=2 assigned_dev_pci_write_config: (4.0): address=0004 > val=0x00000446 len=2 assigned_dev_pci_write_config: NON BAR (4.0): The writing to MMIO have been intercepted, but code fail to count it? Strange... Could you try this debug? memcpy(&msg_ctrl, va + i * 16 + 12, 4); (Further more, you can check the content of msg_ctrl when code try to count them). And also please try the fix for MSI-X table size. Thanks! diff --git a/qemu/hw/device-assignment.c b/qemu/hw/device-assignment.c index 09e54ae..ba31bed 100644 --- a/qemu/hw/device-assignment.c +++ b/qemu/hw/device-assignment.c @@ -45,7 +45,7 @@ #define IORESOURCE_DMA 0x00000800 #define IORESOURCE_PREFETCH 0x00001000 /* No side effects */ -/* #define DEVICE_ASSIGNMENT_DEBUG 1 */ +#define DEVICE_ASSIGNMENT_DEBUG 1 #ifdef DEVICE_ASSIGNMENT_DEBUG #define DEBUG(fmt, ...) \ @@ -816,9 +816,13 @@ static int assigned_dev_update_msix_mmio(PCIDevice *pci_dev) else pos = pci_dev->cap.start; + fprintf(stderr, "the MSIX capabilty position is 0x%lx\n", pos); + entries_max_nr = pci_dev->config[pos + 2]; entries_max_nr &= PCI_MSIX_TABSIZE; + fprintf(stderr, "the MSIX entries_max_nr is 0x%lx\n", entries_max_nr); + /* Get the usable entry number for allocating */ for (i = 0; i < entries_max_nr; i++) {