From patchwork Mon Mar 19 20:06:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: KarimAllah Ahmed X-Patchwork-Id: 10294731 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1D01B60386 for ; Mon, 19 Mar 2018 20:06:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D1592920C for ; Mon, 19 Mar 2018 20:06:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0153C294A4; Mon, 19 Mar 2018 20:06:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 580302920C for ; Mon, 19 Mar 2018 20:06:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031163AbeCSUGW (ORCPT ); Mon, 19 Mar 2018 16:06:22 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:34511 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1031167AbeCSUGR (ORCPT ); Mon, 19 Mar 2018 16:06:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1521489977; x=1553025977; h=from:to:cc:subject:date:message-id; bh=teafAWD3u1y884SMyUKFM7IjctOPsGqh1a3vEgLzrI0=; b=rAarTKDFEqLl++YKN9TkU52hDmKPzo/p/Zlqhlsmigr+52nDWOMzjJBL RuW+E/3oqUOxJn3lbu4aXZuXCBbILh+QSfz8qRoySuEFf9QOueEc1fRBw dracd0OdgHqiKGz0w4QxmsavVTa+hC4bXttSsSjBWpKONJA/AUd2amtQ6 c=; X-IronPort-AV: E=Sophos;i="5.48,331,1517875200"; d="scan'208";a="336517795" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Mar 2018 20:06:15 +0000 Received: from u54e1ad5160425a4b64ea.ant.amazon.com (pdx2-ws-svc-lb17-vlan2.amazon.com [10.247.140.66]) by email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w2JK6BcL080264 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 19 Mar 2018 20:06:12 GMT Received: from u54e1ad5160425a4b64ea.ant.amazon.com (localhost [127.0.0.1]) by u54e1ad5160425a4b64ea.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w2JK69VO018442; Mon, 19 Mar 2018 21:06:10 +0100 Received: (from karahmed@localhost) by u54e1ad5160425a4b64ea.ant.amazon.com (8.15.2/8.15.2/Submit) id w2JK69R9018441; Mon, 19 Mar 2018 21:06:09 +0100 From: KarimAllah Ahmed To: linux-kernel@vger.kernel.org Cc: KarimAllah Ahmed , Bjorn Helgaas , linux-pci@vger.kernel.org, Bjorn Helgaas Subject: [PATCH v4 1/2] PCI/IOV: Use VF0 cached config registers for other VFs Date: Mon, 19 Mar 2018 21:06:00 +0100 Message-Id: <1521489961-18291-1-git-send-email-karahmed@amazon.de> X-Mailer: git-send-email 2.7.4 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Cache some config data from VF0 and use it for all other VFs instead of reading it from the config space of each VF. We assume these items are the same across all associated VFs: Revision ID Class Code Subsystem Vendor ID Subsystem ID This is an optimization when enabling SR-IOV on a device with many VFs. Cc: Bjorn Helgaas Cc: linux-pci@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: KarimAllah Ahmed [bhelgaas: changelog, simplify comments, remove unused "device"] Signed-off-by: Bjorn Helgaas --- v3->v4: - Restructure the code to handle CONFIG_PCI_ATS drivers/pci/iov.c | 42 +++++++++++++++++++++++++++++++++++------- drivers/pci/pci.h | 4 ++++ drivers/pci/probe.c | 47 ++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 81 insertions(+), 12 deletions(-) diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 677924a..30bf8f7 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -114,6 +114,29 @@ resource_size_t pci_iov_resource_size(struct pci_dev *dev, int resno) return dev->sriov->barsz[resno - PCI_IOV_RESOURCES]; } +static void pci_read_vf_config_common(struct pci_dev *virtfn) +{ + struct pci_dev *physfn = virtfn->physfn; + + /* + * Some config registers are the same across all associated VFs. + * Read them once from VF0 so we can skip reading them from the + * other VFs. + * + * PCIe r4.0, sec 9.3.4.1, technically doesn't require all VFs to + * have the same Revision ID and Subsystem ID, but we assume they + * do. + */ + pci_read_config_dword(virtfn, PCI_CLASS_REVISION, + &physfn->sriov->class); + pci_read_config_byte(virtfn, PCI_HEADER_TYPE, + &physfn->sriov->hdr_type); + pci_read_config_word(virtfn, PCI_SUBSYSTEM_VENDOR_ID, + &physfn->sriov->subsystem_vendor); + pci_read_config_word(virtfn, PCI_SUBSYSTEM_ID, + &physfn->sriov->subsystem_device); +} + int pci_iov_add_virtfn(struct pci_dev *dev, int id) { int i; @@ -136,13 +159,17 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) virtfn->devfn = pci_iov_virtfn_devfn(dev, id); virtfn->vendor = dev->vendor; virtfn->device = iov->vf_device; + virtfn->is_virtfn = 1; + virtfn->physfn = pci_dev_get(dev); + + if (id == 0) + pci_read_vf_config_common(virtfn); + rc = pci_setup_device(virtfn); if (rc) - goto failed0; + goto failed1; virtfn->dev.parent = dev->dev.parent; - virtfn->physfn = pci_dev_get(dev); - virtfn->is_virtfn = 1; virtfn->multifunction = 0; for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) { @@ -163,10 +190,10 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) sprintf(buf, "virtfn%u", id); rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf); if (rc) - goto failed1; + goto failed2; rc = sysfs_create_link(&virtfn->dev.kobj, &dev->dev.kobj, "physfn"); if (rc) - goto failed2; + goto failed3; kobject_uevent(&virtfn->dev.kobj, KOBJ_CHANGE); @@ -174,11 +201,12 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) return 0; -failed2: +failed3: sysfs_remove_link(&dev->dev.kobj, buf); +failed2: + pci_stop_and_remove_bus_device(virtfn); failed1: pci_dev_put(dev); - pci_stop_and_remove_bus_device(virtfn); failed0: virtfn_remove_bus(dev->bus, bus); failed: diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index fcd8191..bdb4ba2 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -271,6 +271,10 @@ struct pci_sriov { u16 driver_max_VFs; /* Max num VFs driver supports */ struct pci_dev *dev; /* Lowest numbered PF */ struct pci_dev *self; /* This PF */ + u32 class; /* VF device */ + u8 hdr_type; /* VF header type */ + u16 subsystem_vendor; /* VF subsystem vendor */ + u16 subsystem_device; /* VF subsystem device */ resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ bool drivers_autoprobe; /* Auto probing of VFs by driver */ }; diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index ef53774..21ee1c3 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -1389,6 +1389,43 @@ int pci_cfg_space_size(struct pci_dev *dev) return PCI_CFG_SPACE_SIZE; } +static int pci_cfg_space_class(struct pci_dev *dev) +{ + int class; + +#ifdef CONFIG_PCI_ATS + if (dev->is_virtfn) + return dev->physfn->sriov->class; +#endif + pci_read_config_dword(dev, PCI_CLASS_REVISION, &class); + return class; +} + +static void pci_cfg_space_subsystem(struct pci_dev *dev, u16 *vendor, u16 *device) +{ +#ifdef CONFIG_PCI_ATS + if (dev->is_virtfn) { + *vendor = dev->physfn->sriov->subsystem_vendor; + *device = dev->physfn->sriov->subsystem_device; + return; + } +#endif + pci_read_config_word(dev, PCI_SUBSYSTEM_VENDOR_ID, vendor); + pci_read_config_word(dev, PCI_SUBSYSTEM_ID, device); +} + +static u8 pci_cfg_space_hdr_type(struct pci_dev *dev) +{ + u8 hdr_type; + +#ifdef CONFIG_PCI_ATS + if (dev->is_virtfn) + return dev->physfn->sriov->hdr_type; +#endif + pci_read_config_byte(dev, PCI_HEADER_TYPE, &hdr_type); + return hdr_type; +} + #define LEGACY_IO_RESOURCE (IORESOURCE_IO | IORESOURCE_PCI_FIXED) static void pci_msi_setup_pci_dev(struct pci_dev *dev) @@ -1454,8 +1491,7 @@ int pci_setup_device(struct pci_dev *dev) struct pci_bus_region region; struct resource *res; - if (pci_read_config_byte(dev, PCI_HEADER_TYPE, &hdr_type)) - return -EIO; + hdr_type = pci_cfg_space_hdr_type(dev); dev->sysdata = dev->bus->sysdata; dev->dev.parent = dev->bus->bridge; @@ -1477,7 +1513,8 @@ int pci_setup_device(struct pci_dev *dev) dev->bus->number, PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn)); - pci_read_config_dword(dev, PCI_CLASS_REVISION, &class); + class = pci_cfg_space_class(dev); + dev->revision = class & 0xff; dev->class = class >> 8; /* upper 3 bytes */ @@ -1517,8 +1554,8 @@ int pci_setup_device(struct pci_dev *dev) goto bad; pci_read_irq(dev); pci_read_bases(dev, 6, PCI_ROM_ADDRESS); - pci_read_config_word(dev, PCI_SUBSYSTEM_VENDOR_ID, &dev->subsystem_vendor); - pci_read_config_word(dev, PCI_SUBSYSTEM_ID, &dev->subsystem_device); + + pci_cfg_space_subsystem(dev, &dev->subsystem_vendor, &dev->subsystem_device); /* * Do the ugly legacy mode stuff here rather than broken chip