From patchwork Thu Jan 5 19:28:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Streetman X-Patchwork-Id: 9499613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5EB49606B5 for ; Thu, 5 Jan 2017 19:31:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43D7B28451 for ; Thu, 5 Jan 2017 19:31:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3856828454; Thu, 5 Jan 2017 19:31:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B4D7928451 for ; Thu, 5 Jan 2017 19:31:46 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cPDjW-0006gW-3q; Thu, 05 Jan 2017 19:29:54 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cPDjU-0006fs-P7 for xen-devel@lists.xenproject.org; Thu, 05 Jan 2017 19:29:52 +0000 Received: from [85.158.143.35] by server-7.bemta-6.messagelabs.com id F6/4C-29440-0BE9E685; Thu, 05 Jan 2017 19:29:52 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrELMWRWlGSWpSXmKPExsUSvTeyQHf9vLw Ig4O7JS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1oxDrxaxFGyQqLi15h5rA2OXSBcjF4eQwAIm iVXHtrJ1MXJysAkYSuzZ1sQEYosIGEl03rnMAlLELLCRUWJ63yOwhLBAoMT1I4+YQWwWAVWJJ a3r2UFsXgEHiT+X/oLVSAjISVza9oV5AiPHAkaGVYzqxalFZalFuqZ6SUWZ6RkluYmZObqGBm Z6uanFxYnpqTmJScV6yfm5mxiBfmEAgh2M0y/7H2KU5GBSEuVNnZEXIcSXlJ9SmZFYnBFfVJq TWnyIUYaDQ0mCN2QuUE6wKDU9tSItMwcYIDBpCQ4eJRFeXWCQCPEWFyTmFmemQ6ROMSpKifNO AukTAElklObBtcGC8hKjrJQwLyPQIUI8BalFuZklqPKvGMU5GJWEeQNApvBk5pXATX8FtJgJa PH2gGyQxSWJCCmpBsadJzRzX8TMuyUcuN6w5lnajTenlP0lTx1oL50SV3uk/uD8e6m9DMy9qz Oyoz8FHzBzf+R2o0T2bvth8YRbG67UOqyaG5518f9zd1OLM5HsMr9eiF6py+ZL6lp1l+nE5nW CQlHPTmp1CyU+/l0//cWcD6xL3X1mCpbe4zxc4Pu4Zoru4d/RgtFKLMUZiYZazEXFiQA5bTFd RQIAAA== X-Env-Sender: dan.streetman@canonical.com X-Msg-Ref: server-4.tower-21.messagelabs.com!1483644591!45172014!1 X-Originating-IP: [91.189.89.112] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 18127 invoked from network); 5 Jan 2017 19:29:51 -0000 Received: from youngberry.canonical.com (HELO youngberry.canonical.com) (91.189.89.112) by server-4.tower-21.messagelabs.com with AES256-SHA encrypted SMTP; 5 Jan 2017 19:29:51 -0000 Received: from 45-27-90-188.lightspeed.rlghnc.sbcglobal.net ([45.27.90.188] helo=toughbook.lan) by youngberry.canonical.com with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1cPDjQ-0006i7-K6; Thu, 05 Jan 2017 19:29:48 +0000 From: Dan Streetman To: Konrad Rzeszutek Wilk Date: Thu, 5 Jan 2017 14:28:56 -0500 Message-Id: <20170105192856.25559-1-dan.streetman@canonical.com> X-Mailer: git-send-email 2.9.3 Cc: Bjorn Helgaas , xen-devel@lists.xenproject.org, Dan Streetman , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org Subject: [Xen-devel] [PATCH] xen: do not re-use pirq number cached in pci device msi msg data X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Do not read a pci device's msi message data to see if a pirq was previously configured for the device's msi/msix, as the old pirq was unmapped and may now be in use by another pci device. The previous pirq should never be re-used; instead a new pirq should always be allocated from the hypervisor. The xen_hvm_setup_msi_irqs() function currently checks the pci device's msi descriptor message data for each msi/msix vector it sets up, and if it finds the vector was previously configured with a pirq, and that pirq is mapped to an irq, it re-uses the pirq instead of requesting a new pirq from the hypervisor. However, that pirq was unmapped when the pci device disabled its msi/msix, and it cannot be re-used; it may have been given to a different pci device. This exact situation is happening in a Xen guest where multiple NVMe controllers (pci devices) are present. The NVMe driver configures each pci device's msi/msix twice; first to configure a single vector (to talk to the controller for its configuration info), and then it disables that msi/msix and re-configures with all the msi/msix it needs. When multiple NVMe controllers are present, this happens concurrently on all of them, and in the time between controller A calling pci_disable_msix() and then calling pci_enable_msix_range(), controller B enables its msix and gets controller A's pirq allocated from the hypervisor. Then when controller A re-configures its msix, its first vector tries to re-use the same pirq that it had before; but that pirq was allocated to controller B, and thus the Xen event channel for controller A's re-used pirq fails to map its irq to that pirq; the hypervisor already has the pirq mapped elsewhere. Signed-off-by: Dan Streetman --- arch/x86/pci/xen.c | 23 +++++++---------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c index bedfab9..a00a6c0 100644 --- a/arch/x86/pci/xen.c +++ b/arch/x86/pci/xen.c @@ -234,23 +234,14 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type) return 1; for_each_pci_msi_entry(msidesc, dev) { - __pci_read_msi_msg(msidesc, &msg); - pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) | - ((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff); - if (msg.data != XEN_PIRQ_MSI_DATA || - xen_irq_from_pirq(pirq) < 0) { - pirq = xen_allocate_pirq_msi(dev, msidesc); - if (pirq < 0) { - irq = -ENODEV; - goto error; - } - xen_msi_compose_msg(dev, pirq, &msg); - __pci_write_msi_msg(msidesc, &msg); - dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); - } else { - dev_dbg(&dev->dev, - "xen: msi already bound to pirq=%d\n", pirq); + pirq = xen_allocate_pirq_msi(dev, msidesc); + if (pirq < 0) { + irq = -ENODEV; + goto error; } + xen_msi_compose_msg(dev, pirq, &msg); + __pci_write_msi_msg(msidesc, &msg); + dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq); irq = xen_bind_pirq_msi_to_irq(dev, msidesc, pirq, (type == PCI_CAP_ID_MSI) ? nvec : 1, (type == PCI_CAP_ID_MSIX) ?