From patchwork Mon Jun 18 22:01:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 10472743 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D62BD6020C for ; Mon, 18 Jun 2018 22:02:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C586F28771 for ; Mon, 18 Jun 2018 22:02:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BA2F7288BD; Mon, 18 Jun 2018 22:02:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 403AF28771 for ; Mon, 18 Jun 2018 22:02:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755346AbeFRWC3 (ORCPT ); Mon, 18 Jun 2018 18:02:29 -0400 Received: from mail-wr0-f194.google.com ([209.85.128.194]:34486 "EHLO mail-wr0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755344AbeFRWC2 (ORCPT ); Mon, 18 Jun 2018 18:02:28 -0400 Received: by mail-wr0-f194.google.com with SMTP id a12-v6so18411588wro.1 for ; Mon, 18 Jun 2018 15:02:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=WGNSrcU0tPUaRq7cJ5pmTLwKMxtkll7TX5BvmMkVMZM=; b=H+QIMyEzu0BlcmjpqJ9rKHtKAyFKD5xo/3CMPQe+uT3+rZvxNOEFrVzuXbiJpY15BC o3AKWPAw3e4bpCbE+1PTOGVWuP0x0vRkqBg4+ttRnAOjv3cclvh4885rYLIxIayxa9rn Ua9CylDbHoccGZdOSG1LRxixZFIMGd8O5KjPZ5ANuZErluTT3HW3JKetmNih3ECwj1W/ 8AVZenYeaWV6E866HPWrWYQDwSck9+nFY2VyfS+UH4d65Sbf/h8iE9GIXCp0wx0e0ZfC GkdkAKQfo6veNfpvQ70CFlArGJ+FP7OKCeL7k6uB4hCCmKWdZXnkqreRSXXlkLpgZHkh pEDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=WGNSrcU0tPUaRq7cJ5pmTLwKMxtkll7TX5BvmMkVMZM=; b=NyHAh3tJkfcIR+FcP10d2tK5LrJ6ByVrspHW1Ojl0uO8KwkiuJNYPfNbIyNlUEnNu1 IQ2/FmM5DsKGSx9KJhvkCs4Dw/68ksRGPziGYZN+VwA7Qfw0WgNHpl5vtAu4YciZrYO+ PB0sH87LJppaUrl24COjINEaeucraiuGR0AA1dBcLMAnYEdjxXTfwjiGnscrJ475BC9R moWE/O2WNVZHZoYPrAYcHYWmD/bEYRm9ZqHhcrrPpyYWVH9LujlPeElKNnE1XYrmtD8h 8maqHP58y0oHxe7/9w+9A7r9FbWv57fYx198yuIH3pKPdHMc7WN6OBaZeTW5hAXsYrg+ SaGQ== X-Gm-Message-State: APt69E07FN0KA/ABmqJWs2ZUyHffhQ7YgSwWfTWgY+1c4wDkD6a4VDuq VPf3Qogwn3z4kH8/4JPuudEuKg== X-Google-Smtp-Source: ADUXVKJD7saa/aNgNzqetgZVQF5lfrWuj8ZmIj9SBLvEbpU2LvcZEsM6CHdf8jMJpdK2wse3opxk2w== X-Received: by 2002:adf:d10a:: with SMTP id a10-v6mr11243612wri.18.1529359347079; Mon, 18 Jun 2018 15:02:27 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([75.53.12.129]) by smtp.gmail.com with ESMTPSA id s9-v6sm7085551wmc.34.2018.06.18.15.02.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Jun 2018 15:02:26 -0700 (PDT) From: Jakub Kicinski To: Bjorn Helgaas Cc: linux-pci@vger.kernel.org, alexander.duyck@gmail.com, oss-drivers@netronome.com, Christoph Hellwig , Don Dutile , Jakub Kicinski Subject: [PATCH] PCI: reset driver SR-IOV state after remove Date: Mon, 18 Jun 2018 15:01:42 -0700 Message-Id: <20180618220142.16527-1-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.17.1 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Bjorn points out that currently core and most of the drivers don't clean up dev->sriov->driver_max_VFs settings on .remove(). This means that if a different driver is bound afterwards it will inherit the old setting: - load PF driver 1 - driver 1 calls pci_sriov_set_totalvfs() to reduce driver_max_VFs - unload PF driver 1 - load PF driver 2 # driver 2 does not know to call pci_sriov_set_totalvfs() Some drivers (e.g. nfp) used to do the clean up by calling pci_sriov_set_totalvfs(dev, 0), since 0 was equivalent to no driver limit set. After commit 8d85a7a4f2c9 ("PCI/IOV: Allow PF drivers to limit total_VFs to 0") 0 no longer has this special meaning. The need to reset driver_max_VFs comes from the fact that old FW builds may not expose its VF limits to the drivers, and depend on the ability to reject the configuration change when driver notifies the FW as part of struct pci_driver->sriov_configure() callback. Therefore if there is no information on VF count limits we should use the PCI capability max, and not the last value set by pci_sriov_set_totalvfs(). Reset driver_max_VFs back to total_VFs after device remove. If drivers want to reload FW/reconfigure the device while driver is bound we may add an explicit pci_sriov_reset_totalvfs(), but for now no driver is doing that. Fixes: 8d85a7a4f2c9 ("PCI/IOV: Allow PF drivers to limit total_VFs to 0") Signed-off-by: Jakub Kicinski --- drivers/pci/iov.c | 16 ++++++++++++++++ drivers/pci/pci-driver.c | 1 + drivers/pci/pci.h | 4 ++++ 3 files changed, 21 insertions(+) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index d0d73dbbd5ca..cbbe6d8fab0c 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -574,6 +574,22 @@ void pci_iov_release(struct pci_dev *dev) sriov_release(dev); } +/** + * pci_iov_device_removed - clean up SR-IOV state after PF driver is detached + * @dev: the PCI device + */ +void pci_iov_device_removed(struct pci_dev *dev) +{ + struct pci_sriov *iov = dev->sriov; + + if (!dev->is_physfn) + return; + iov->driver_max_VFs = iov->total_VFs; + if (iov->num_VFs) + dev_warn(&dev->dev, + "driver left SR-IOV enabled after remove\n"); +} + /** * pci_iov_update_resource - update a VF BAR * @dev: the PCI device diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index c125d53033c6..80a281cf5d21 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -445,6 +445,7 @@ static int pci_device_remove(struct device *dev) } pcibios_free_irq(pci_dev); pci_dev->driver = NULL; + pci_iov_device_removed(pci_dev); } /* Undo the runtime PM settings in local_pci_probe() */ diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index c358e7a07f3f..fc8bd4fdfb95 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -311,6 +311,7 @@ static inline void pci_restore_ats_state(struct pci_dev *dev) #ifdef CONFIG_PCI_IOV int pci_iov_init(struct pci_dev *dev); void pci_iov_release(struct pci_dev *dev); +void pci_iov_device_removed(struct pci_dev *dev); void pci_iov_update_resource(struct pci_dev *dev, int resno); resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno); void pci_restore_iov_state(struct pci_dev *dev); @@ -323,6 +324,9 @@ static inline int pci_iov_init(struct pci_dev *dev) } static inline void pci_iov_release(struct pci_dev *dev) +{ +} +static inline void pci_iov_device_removed(struct pci_dev *dev) { } static inline void pci_restore_iov_state(struct pci_dev *dev)