From patchwork Fri Jan 13 20:07:51 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Streetman X-Patchwork-Id: 9516359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A589B601E5 for ; Fri, 13 Jan 2017 20:10:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 991152877D for ; Fri, 13 Jan 2017 20:10:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D99A28782; Fri, 13 Jan 2017 20:10:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1378F2877D for ; Fri, 13 Jan 2017 20:10:55 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cS89C-0000CF-JD; Fri, 13 Jan 2017 20:08:26 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cS89B-0000C6-Ap for xen-devel@lists.xenproject.org; Fri, 13 Jan 2017 20:08:25 +0000 Received: from [85.158.143.35] by server-9.bemta-6.messagelabs.com id 6B/B4-27165-8B339785; Fri, 13 Jan 2017 20:08:24 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrHIsWRWlGSWpSXmKPExsVyMfTGEd3txpU RBr9mmll83zKZyYHR4/CHKywBjFGsmXlJ+RUJrBk9l5ayFRwUrzix7DZjA+NO4S5GLg4hgemM EqsfLmAGcVgEnrJI/P3+mx3EkRDoZpU4ee8uUxcjJ5ATI7H4zDu2LkYOILtSYv3vJJCwkICix NlZU5khJi1hknjZ/pkNJCEsoCVxbMIBdhCbTUBdYtGrb6wgRSICCxklWqbMYwFxmAWeMEocef WcEaIjUOL6kUfMIDaLgKrEl+MLwDbzCphJNCyfzAJxhZzEpW1fwGo4Bewkzvy7yzKBUWABI8M qRo3i1KKy1CJdIzO9pKLM9IyS3MTMHF1DAzO93NTi4sT01JzEpGK95PzcTYzA8GIAgh2MZxYE HmKU5GBSEuX9rloRIcSXlJ9SmZFYnBFfVJqTWnyIUYaDQ0mCd5FRZYSQYFFqempFWmYOMNBh0 hIcPEoivLtB0rzFBYm5xZnpEKlTjJYcPV2nXzJxnLoBIvfsuvySSYglLz8vVUqcdyVIgwBIQ0 ZpHtw4WDReYpSVEuZlBDpQiKcgtSg3swRV/hWjOAejkjBvPcgUnsy8Eritr4AOYgI66KJNOch BJYkIKakGxrTC8n1627KuzcxnWdZkv0iTpefVRnXFNzG607L3FDc07A078vFJYPW0pBU39jX8 /F8uvELwpFOxXxf7k5zuI6mbZ+mpaH4UEa19fcM69+B9AfftFWq/fRp0XMxqBA9evmZed837g prCPP7ZC+5czb+usky29KT/4+OyfV27qjybbqzY7nk7UYmlOCPRUIu5qDgRAJSmITDBAgAA X-Env-Sender: ddstreet@gmail.com X-Msg-Ref: server-11.tower-21.messagelabs.com!1484338103!52524617!1 X-Originating-IP: [209.85.216.196] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 59229 invoked from network); 13 Jan 2017 20:08:23 -0000 Received: from mail-qt0-f196.google.com (HELO mail-qt0-f196.google.com) (209.85.216.196) by server-11.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 13 Jan 2017 20:08:23 -0000 Received: by mail-qt0-f196.google.com with SMTP id f4so7515472qte.2 for ; Fri, 13 Jan 2017 12:08:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to; bh=XpjFgfj14UwIriQlQRKvj5AP9b78pYuUam8LX9590jQ=; b=QPo98ykZTlHsCNDX9GnkmOblxb0oiuxYv2OrVyAkJBIKP6VtiBtLINBvO7xq67jlvH 4sqed+FfvUd4VJqVj6+R6ORbS3VmNqTrKQR5kVwNoGyIdH3aBf6cGAyEaA2XvpBeFJ6l yiWCwF4khFwOal0dkyV0LB0c1wxVJWAtvbCKtulXNQPKiTRg93vp20s9wWlW0cAw0l6y a0LodZu19QOUoeqZK/HHGK+8KE/44Pk67ia1AoQ+LCkH+e26sexB0FchnZn7Ifh7c4yr /4LT7w1ylWVFtNlqaKkV+uWcHCNnKGHyiPBC8dc/84zngvXosSqYNjjG5l2oaxc3qfsL 6gMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to; bh=XpjFgfj14UwIriQlQRKvj5AP9b78pYuUam8LX9590jQ=; b=U5Sk93sVkRpsIg9y71VZzjQC3bHAOShJbxghOVadCm1vL02GxV4oZN7kIBNuFfH+mM x4Nu2f4QeNFSrBuu5NQiBLnKwhK18HU0DN/uflx2XOLDcugnBDOdgt4EK+09dOIZTJlT dyZLt/U21C3uMMqASaKPv3h0egRO6V+j6jfpAnH+7EUYcbHBPkBCw6SvdTCEHLaLPw3C qn3ZP++pIcX3hhYXA8M3W5IcIis19Ld9ZgPEGtkXxVJ1lET+nNm5yrgGxSLK1/wpo6um DPUAmrBKvAX231YQITylvwGMCOyAqYbFLT66mNP7vTUnaL+Phf9X0dgQPwp71qZnWxnP LWmw== X-Gm-Message-State: AIkVDXIDk+fXrAwYtYr8O80V2mqeteAkq8bGxN3YI+969mXVLSPQvrHThdjl2OJESngKXA== X-Received: by 10.237.34.150 with SMTP id p22mr2259228qtc.76.1484338102031; Fri, 13 Jan 2017 12:08:22 -0800 (PST) Received: from toughbook.lan (45-27-90-188.lightspeed.rlghnc.sbcglobal.net. [45.27.90.188]) by smtp.gmail.com with ESMTPSA id b12sm9975088qta.27.2017.01.13.12.08.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Jan 2017 12:08:21 -0800 (PST) From: Dan Streetman To: Stefano Stabellini , jgross@suse.com, Konrad Rzeszutek Wilk , Boris Ostrovsky Date: Fri, 13 Jan 2017 15:07:51 -0500 Message-Id: <20170113200751.20125-1-ddstreet@ieee.org> X-Mailer: git-send-email 2.9.3 In-Reply-To: <04f2a09f-59be-a720-bc98-4afb53171790@oracle.com> Cc: linux-pci@vger.kernel.org, linux-kernel , Bjorn Helgaas , xen-devel@lists.xenproject.org, Dan Streetman , Dan Streetman Subject: [Xen-devel] [PATCH] xen: do not re-use pirq number cached in pci device msi msg data X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Revert the main part of commit: af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests") That commit introduced reading the pci device's msi message data to see if a pirq was previously configured for the device's msi/msix, and re-use that pirq. At the time, that was the correct behavior. However, a later change to Qemu caused it to call into the Xen hypervisor to unmap all pirqs for a pci device, when the pci device disables its MSI/MSIX vectors; specifically the Qemu commit: c976437c7dba9c7444fb41df45468968aaa326ad ("qemu-xen: free all the pirqs for msi/msix when driver unload") Once Qemu added this pirq unmapping, it was no longer correct for the kernel to re-use the pirq number cached in the pci device msi message data. All Qemu releases since 2.1.0 contain the patch that unmaps the pirqs when the pci device disables its MSI/MSIX vectors. This bug is causing failures to initialize multiple NVMe controllers under Xen, because the NVMe driver sets up a single MSIX vector for each controller (concurrently), and then after using that to talk to the controller for some configuration data, it disables the single MSIX vector and re-configures all the MSIX vectors it needs. So the MSIX setup code tries to re-use the cached pirq from the first vector for each controller, but the hypervisor has already given away that pirq to another controller, and its initialization fails. This is discussed in more detail at: https://lists.xen.org/archives/html/xen-devel/2017-01/msg00447.html Fixes: af42b8d12f8a ("xen: fix MSI setup and teardown for PV on HVM guests") Signed-off-by: Dan Streetman Reviewed-by: Stefano Stabellini Reviewed-by: Boris Ostrovsky Acked-by: Konrad Rzeszutek Wilk --- arch/x86/pci/xen.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index bedfab9..a00a6c0 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -234,23 +234,14 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) return 1; for_each_pci_msi_entry(msidesc, dev) { - __pci_read_msi_msg(msidesc, &msg); - pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | - ((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff); - if (msg.data != XEN_PIRQ_MSI_DATA || - xen_irq_from_pirq(pirq) < 0) { - pirq = xen_allocate_pirq_msi(dev, msidesc); - if (pirq < 0) { - irq = -ENODEV; - goto error; - } - xen_msi_compose_msg(dev, pirq, &msg); - __pci_write_msi_msg(msidesc, &msg); - dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); - } else { - dev_dbg(&dev->dev, - "xen: msi already bound to pirq=%d\n", pirq); + pirq = xen_allocate_pirq_msi(dev, msidesc); + if (pirq < 0) { + irq = -ENODEV; + goto error; } + xen_msi_compose_msg(dev, pirq, &msg); + __pci_write_msi_msg(msidesc, &msg); + dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); irq = xen_bind_pirq_msi_to_irq(dev, msidesc, pirq, (type == PCI_CAP_ID_MSI) ? nvec : 1, (type == PCI_CAP_ID_MSIX) ?