From patchwork Fri Oct 20 16:22:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 10020613 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 28297602CB for ; Fri, 20 Oct 2017 16:22:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1DDB328996 for ; Fri, 20 Oct 2017 16:22:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 12E4028F10; Fri, 20 Oct 2017 16:22:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A509528996 for ; Fri, 20 Oct 2017 16:22:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752854AbdJTQWc (ORCPT ); Fri, 20 Oct 2017 12:22:32 -0400 Received: from mail.kernel.org ([198.145.29.99]:47420 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752639AbdJTQWa (ORCPT ); Fri, 20 Oct 2017 12:22:30 -0400 Received: from localhost (unknown [69.71.4.159]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 71ECA2191E; Fri, 20 Oct 2017 16:22:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71ECA2191E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=helgaas@kernel.org Subject: [PATCH v6 4/4] PCI/portdrv: Compute MSI/MSI-X IRQ vectors after final allocation From: Bjorn Helgaas To: Dongdong Liu Cc: Charles Chenxin , linuxarm@huawei.com, Christoph Hellwig , Gabriele Paoloni , linux-pci@vger.kernel.org Date: Fri, 20 Oct 2017 11:22:27 -0500 Message-ID: <20171020162227.6391.14432.stgit@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: <20171020161922.6391.9091.stgit@bhelgaas-glaptop.roam.corp.google.com> References: <20171020161922.6391.9091.stgit@bhelgaas-glaptop.roam.corp.google.com> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Bjorn Helgaas When setting up portdrv MSI/MSI-X interrupts, we previously allocated the maximum possible number of vectors, read the Interrupt Message Numbers for each service, saved the IRQ for each, freed the vectors, and finally used the largest Message Number to reallocate only as many vectors as we need. The problem is that freeing the vectors invalidates their IRQs, so the saved IRQ numbers may now be invalid, which can result in errors like this: pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22 pciehp 0000:00:00.0:pcie004: Cannot get irq 20 for the hotplug controller aer: probe of 0000:00:00.0:pcie002 failed with error -22 dpc 0000:00:00.0:pcie010: request IRQ22 failed: -22 Change the setup so we save the Interrupt Message Numbers (not the IRQs) before we free the original setup, then use the Message Numbers to compute the IRQs (via pci_irq_vector()) *after* we reallocate the vectors. This should always be safe for MSI-X because the Message Numbers are fixed. For MSI, the hardware is allowed to change Message Numbers when we update the MSI Multiple Message Enable field when reallocating the vectors, but since we allocate enough vectors to accommodate the largest Message Number we found, that's unlikely. See PCIe r3.1, sec 7.8.2, 7.10.10, 7.31.2. Fixes: 3674cc49da9a ("PCI/portdrv: Use pci_irq_alloc_vectors()") Based-on-patch-by: Dongdong Liu Tested-by: Dongdong Liu # HiSilicon hip08 Signed-off-by: Bjorn Helgaas Reviewed-by: Christoph Hellwig --- drivers/pci/pcie/portdrv_core.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/drivers/pci/pcie/portdrv_core.c b/drivers/pci/pcie/portdrv_core.c index 102f5c5fcfe0..3cd5eb48644a 100644 --- a/drivers/pci/pcie/portdrv_core.c +++ b/drivers/pci/pcie/portdrv_core.c @@ -117,6 +117,26 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask) return -EIO; } + /* + * If we allocated more than we need, free them and reallocate fewer. + * + * Reallocating may change the specific vectors we get, so + * pci_irq_vector() must be done *after* the reallocation. + * + * If we're using MSI, hardware is *allowed* to change the Interrupt + * Message Numbers when we free and reallocate the vectors, but we + * assume it won't because we allocate enough vectors for the + * biggest Message Number we found. + */ + if (nvec != nr_entries) { + pci_free_irq_vectors(dev); + + nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, + PCI_IRQ_MSIX | PCI_IRQ_MSI); + if (nr_entries < 0) + return nr_entries; + } + /* PME and hotplug share an MSI/MSI-X vector */ if (mask & (PCIE_PORT_SERVICE_PME | PCIE_PORT_SERVICE_HP)) { irqs[PCIE_PORT_SERVICE_PME_SHIFT] = pci_irq_vector(dev, pme); @@ -129,16 +149,6 @@ static int pcie_port_enable_irq_vec(struct pci_dev *dev, int *irqs, int mask) if (mask & PCIE_PORT_SERVICE_DPC) irqs[PCIE_PORT_SERVICE_DPC_SHIFT] = pci_irq_vector(dev, dpc); - /* If we allocated more than we need, free them and allocate fewer */ - if (nvec != nr_entries) { - pci_free_irq_vectors(dev); - - nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec, - PCI_IRQ_MSIX | PCI_IRQ_MSI); - if (nr_entries < 0) - return nr_entries; - } - return 0; }