From patchwork Wed Aug 30 19:02:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 9930663 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D55B16022E for ; Wed, 30 Aug 2017 19:02:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C641628761 for ; Wed, 30 Aug 2017 19:02:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BAE1428764; Wed, 30 Aug 2017 19:02:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D80228761 for ; Wed, 30 Aug 2017 19:02:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750803AbdH3TCJ (ORCPT ); Wed, 30 Aug 2017 15:02:09 -0400 Received: from mail.kernel.org ([198.145.29.99]:57850 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750757AbdH3TCI (ORCPT ); Wed, 30 Aug 2017 15:02:08 -0400 Received: from localhost (unknown [69.55.156.165]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 333DB21A92; Wed, 30 Aug 2017 19:02:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 333DB21A92 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=helgaas@kernel.org Date: Wed, 30 Aug 2017 14:02:06 -0500 From: Bjorn Helgaas To: Alexey Kardashevskiy Cc: linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, Yongji Xie , Eric Auger , Bjorn Helgaas , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, shan.gavin@gmail.com, Benjamin Herrenschmidt , Paul Mackerras , Gavin Shan Subject: Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable() Message-ID: <20170830190206.GU8154@bhelgaas-glaptop.roam.corp.google.com> References: <20170811081933.17474-1-aik@ozlabs.ru> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20170811081933.17474-1-aik@ozlabs.ru> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Aug 11, 2017 at 06:19:33PM +1000, Alexey Kardashevskiy wrote: > From: Gavin Shan > > The PowerNV platform is the only user of pcibios_sriov_disable(). > The IOV BAR could be shifted by pci_iov_update_resource(). The > warning message in the function is printed if the IOV capability > is in enabled (PCI_SRIOV_CTRL_VFE && PCI_SRIOV_CTRL_MSE) state. > > This is the backtrace of what is happening: > pci_disable_sriov > sriov_disable > pnv_pci_sriov_disable > pnv_pci_vf_resource_shift > pci_update_resource > pci_iov_update_resource > > This fixes the issue by disabling IOV capability before calling > pcibios_sriov_disable(). With it, the disabling path matches > the enabling path: pcibios_sriov_enable() is called before the > IOV capability is enabled. > > Cc: shan.gavin@gmail.com > Cc: Benjamin Herrenschmidt > Cc: Paul Mackerras > Reported-by: Carol L Soto > Signed-off-by: Gavin Shan > Tested-by: Carol L Soto > Signed-off-by: Alexey Kardashevskiy > --- > > This is repost. Since Gavin left the team, I am trying to push it out. > The previos converstion is here: https://patchwork.ozlabs.org/patch/732653/ I gave up on the previous issue. I think this patch makes sense as-is at least as far as the fact that we can't update a struct resource while the device is still consuming it. I reworked the changelog to emphasize that. I assume the fact that pci_iov_update_resource() dropped the resource update caused some user-visible issue later on, and I might mention that, too, if I knew what it was. Here's what I would consider putting on pci/virtualization (the diff is unchanged from your post): commit 08132e7759b3929bea0ccdf8afe81ebf05351389 Author: Gavin Shan Date: Fri Aug 11 18:19:33 2017 +1000 PCI: Disable VF decoding before updating resources in pcibios_sriov_disable() A struct resource represents the address space consumed by a device. We should not modify that resource while the device is actively using the address space. For VFs, pci_iov_update_resource() enforces this by printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF Memory Space Enable) bits are set. Previously, both sriov_enable() and sriov_disable() called the pcibios_sriov_disable() arch hook, which may update the struct resource, while VFE and MSE were enabled. This effectively dropped the resource update pcibios_sriov_disable() intended to do. Disable VF memory decoding before calling pcibios_sriov_disable(). Reported-by: Carol L Soto Tested-by: Carol L Soto Signed-off-by: Gavin Shan Signed-off-by: Alexey Kardashevskiy [bhelgaas: changelog] Signed-off-by: Bjorn Helgaas Cc: shan.gavin@gmail.com Cc: Benjamin Herrenschmidt Cc: Paul Mackerras diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 120485d6f352..ac41c8be9200 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -331,7 +331,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) while (i--) pci_iov_remove_virtfn(dev, i, 0); - pcibios_sriov_disable(dev); err_pcibios: iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE); pci_cfg_access_lock(dev); @@ -339,6 +338,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn) ssleep(1); pci_cfg_access_unlock(dev); + pcibios_sriov_disable(dev); + if (iov->link != dev->devfn) sysfs_remove_link(&dev->dev.kobj, "dep_link"); @@ -357,14 +358,14 @@ static void sriov_disable(struct pci_dev *dev) for (i = 0; i < iov->num_VFs; i++) pci_iov_remove_virtfn(dev, i, 0); - pcibios_sriov_disable(dev); - iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE); pci_cfg_access_lock(dev); pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl); ssleep(1); pci_cfg_access_unlock(dev); + pcibios_sriov_disable(dev); + if (iov->link != dev->devfn) sysfs_remove_link(&dev->dev.kobj, "dep_link");