From patchwork Wed Oct 16 17:04:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 11194577 X-Patchwork-Delegate: lorenzo.pieralisi@arm.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 92E3417E6 for ; Wed, 16 Oct 2019 23:06:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7C3CB218DE for ; Wed, 16 Oct 2019 23:06:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731616AbfJPXGZ (ORCPT ); Wed, 16 Oct 2019 19:06:25 -0400 Received: from mga02.intel.com ([134.134.136.20]:4571 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390020AbfJPXGY (ORCPT ); Wed, 16 Oct 2019 19:06:24 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Oct 2019 16:06:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,305,1566889200"; d="scan'208";a="189833614" Received: from unknown (HELO nsgsw-rhel7p6.lm.intel.com) ([10.232.116.93]) by orsmga008.jf.intel.com with ESMTP; 16 Oct 2019 16:06:23 -0700 From: Jon Derrick To: Lorenzo Pieralisi Cc: Bjorn Helgaas , , Pawel Baldysiak , Artur Paszkiewicz , Keith Busch , Dave Fugate , Andy Shevchenko , Jon Derrick Subject: [PATCH 1/3] PCI: vmd: Add helpers to access device config space Date: Wed, 16 Oct 2019 11:04:46 -0600 Message-Id: <1571245488-3549-2-git-send-email-jonathan.derrick@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> References: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org This patch adds helpers to access child device config space. It uses the fabric-view of the bus number, which requires the pci accessors to translate out the starting bus number. This will allow internal code to access child device config space without a struct pci_bus while minding the accessing rules. Signed-off-by: Jon Derrick Reviewed-by: Andy Shevchenko --- drivers/pci/controller/vmd.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index a35d3f3..959c7c7 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -440,12 +440,10 @@ static void vmd_setup_dma_ops(struct vmd_dev *vmd) } #undef ASSIGN_VMD_DMA_OPS -static char __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus, +static char __iomem *vmd_cfg_addr(struct vmd_dev *vmd, unsigned char busn, unsigned int devfn, int reg, int len) { - char __iomem *addr = vmd->cfgbar + - ((bus->number - vmd->busn_start) << 20) + - (devfn << 12) + reg; + char __iomem *addr = vmd->cfgbar + (busn << 20) + (devfn << 12) + reg; if ((addr - vmd->cfgbar) + len >= resource_size(&vmd->dev->resource[VMD_CFGBAR])) @@ -458,11 +456,10 @@ static char __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus, * CPU may deadlock if config space is not serialized on some versions of this * hardware, so all config space access is done under a spinlock. */ -static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg, - int len, u32 *value) +static int vmd_cfg_read(struct vmd_dev *vmd, unsigned char busn, + unsigned int devfn, int reg, int len, u32 *value) { - struct vmd_dev *vmd = vmd_from_bus(bus); - char __iomem *addr = vmd_cfg_addr(vmd, bus, devfn, reg, len); + char __iomem *addr = vmd_cfg_addr(vmd, busn, devfn, reg, len); unsigned long flags; int ret = 0; @@ -488,16 +485,23 @@ static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg, return ret; } +static int vmd_pci_read(struct pci_bus *bus, unsigned int devfn, int reg, + int len, u32 *value) +{ + struct vmd_dev *vmd = vmd_from_bus(bus); + return vmd_cfg_read(vmd, bus->number - vmd->busn_start, devfn, + reg, len, value); +} + /* * VMD h/w converts non-posted config writes to posted memory writes. The * read-back in this function forces the completion so it returns only after * the config space was written, as expected. */ -static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, - int len, u32 value) +static int vmd_cfg_write(struct vmd_dev *vmd, unsigned char busn, + unsigned int devfn, int reg, int len, u32 value) { - struct vmd_dev *vmd = vmd_from_bus(bus); - char __iomem *addr = vmd_cfg_addr(vmd, bus, devfn, reg, len); + char __iomem *addr = vmd_cfg_addr(vmd, busn, devfn, reg, len); unsigned long flags; int ret = 0; @@ -526,6 +530,14 @@ static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, return ret; } +static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, + int len, u32 value) +{ + struct vmd_dev *vmd = vmd_from_bus(bus); + return vmd_cfg_write(vmd, bus->number - vmd->busn_start, devfn, + reg, len, value); +} + static struct pci_ops vmd_ops = { .read = vmd_pci_read, .write = vmd_pci_write, From patchwork Wed Oct 16 17:04:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 11194579 X-Patchwork-Delegate: lorenzo.pieralisi@arm.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E58DC76 for ; Wed, 16 Oct 2019 23:06:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CED9121925 for ; Wed, 16 Oct 2019 23:06:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2405788AbfJPXG0 (ORCPT ); Wed, 16 Oct 2019 19:06:26 -0400 Received: from mga02.intel.com ([134.134.136.20]:4571 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390020AbfJPXG0 (ORCPT ); Wed, 16 Oct 2019 19:06:26 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Oct 2019 16:06:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,305,1566889200"; d="scan'208";a="189833617" Received: from unknown (HELO nsgsw-rhel7p6.lm.intel.com) ([10.232.116.93]) by orsmga008.jf.intel.com with ESMTP; 16 Oct 2019 16:06:24 -0700 From: Jon Derrick To: Lorenzo Pieralisi Cc: Bjorn Helgaas , , Pawel Baldysiak , Artur Paszkiewicz , Keith Busch , Dave Fugate , Andy Shevchenko , Jon Derrick Subject: [PATCH 2/3] PCI: vmd: Expose VMD details from BIOS Date: Wed, 16 Oct 2019 11:04:47 -0600 Message-Id: <1571245488-3549-3-git-send-email-jonathan.derrick@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> References: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When some VMDs are enabled and others are not, it's difficult to determine which IIO stack corresponds to the enabled VMD. To assist userspace with management tasks, VMD BIOS will write the VMD instance number and socket number into the first enabled root port's IO Base/Limit registers prior to OS handoff. VMD driver can capture this information and expose it to userspace. Signed-off-by: Jon Derrick Reviewed-by: Andy Shevchenko --- drivers/pci/controller/vmd.c | 79 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 77 insertions(+), 2 deletions(-) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index 959c7c7..dbe1bff 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -98,6 +98,8 @@ struct vmd_dev { struct irq_domain *irq_domain; struct pci_bus *bus; u8 busn_start; + u8 socket_nr; + u8 instance_nr; struct dma_map_ops dma_ops; struct dma_domain dma_domain; @@ -543,6 +545,74 @@ static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, .write = vmd_pci_write, }; +/** + * for_each_vmd_root_port - iterate over all enabled VMD Root Ports + * @vmd: &struct vmd_dev VMD device descriptor + * @rp: int iterator cursor + * @temp: u32 temporary value for config read + * + * VMD Root Ports are located in the VMD PCIe Domain at 00:[0-3].0, and config + * space can be determinately accessed through the VMD Config BAR. Because VMD + * Root Ports can be individually disabled, it's important to iterate for the + * first enabled Root Port as determined by reading the Vendor/Device register. + */ +#define for_each_vmd_root_port(vmd, rp, temp) \ + for (rp = 0; rp < 4; rp++) \ + if (vmd_cfg_read(vmd, 0, PCI_DEVFN(root_port, 0), \ + PCI_VENDOR_ID, 4, &temp) || \ + temp == 0xffffffff) {} else + +static int vmd_parse_domain(struct vmd_dev *vmd) +{ + int root_port, ret; + u32 temp, iobase; + + vmd->socket_nr = -1; + vmd->instance_nr = -1; + + for_each_vmd_root_port(vmd, root_port, temp) { + ret = vmd_cfg_read(vmd, 0, PCI_DEVFN(root_port, 0), + PCI_IO_BASE, 2, &iobase); + if (ret) + return ret; + + vmd->socket_nr = (iobase >> 4) & 0xf; + vmd->instance_nr = (iobase >> 14) & 0x3; + + /* First available will be used */ + break; + } + + return 0; +} + +static ssize_t socket_nr_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct vmd_dev *vmd = pci_get_drvdata(pdev); + + return sprintf(buf, "%u\n", vmd->socket_nr); +} +static DEVICE_ATTR_RO(socket_nr); + +static ssize_t instance_nr_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + struct vmd_dev *vmd = pci_get_drvdata(pdev); + + return sprintf(buf, "%u\n", vmd->instance_nr); +} +static DEVICE_ATTR_RO(instance_nr); + +static struct attribute *vmd_dev_attrs[] = { + &dev_attr_socket_nr.attr, + &dev_attr_instance_nr.attr, + NULL +}; +ATTRIBUTE_GROUPS(vmd_dev); + static void vmd_attach_resources(struct vmd_dev *vmd) { vmd->dev->resource[VMD_MEMBAR1].child = &vmd->resources[1]; @@ -582,6 +652,11 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) resource_size_t offset[2] = {0}; resource_size_t membar2_offset = 0x2000; struct pci_bus *child; + int ret; + + ret = vmd_parse_domain(vmd); + if (ret) + return ret; /* * Shadow registers may exist in certain VMD device ids which allow @@ -591,7 +666,6 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) */ if (features & VMD_FEAT_HAS_MEMBAR_SHADOW) { u32 vmlock; - int ret; membar2_offset = MB2_SHADOW_OFFSET + MB2_SHADOW_SIZE; ret = pci_read_config_dword(vmd->dev, PCI_REG_VMLOCK, &vmlock); @@ -876,7 +950,8 @@ static int vmd_resume(struct device *dev) .probe = vmd_probe, .remove = vmd_remove, .driver = { - .pm = &vmd_dev_pm_ops, + .pm = &vmd_dev_pm_ops, + .dev_groups = vmd_dev_groups, }, }; module_pci_driver(vmd_drv); From patchwork Wed Oct 16 17:04:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jon Derrick X-Patchwork-Id: 11194581 X-Patchwork-Delegate: lorenzo.pieralisi@arm.com Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 31C0E14DB for ; Wed, 16 Oct 2019 23:06:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0E0F6207FF for ; Wed, 16 Oct 2019 23:06:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390020AbfJPXG0 (ORCPT ); Wed, 16 Oct 2019 19:06:26 -0400 Received: from mga02.intel.com ([134.134.136.20]:4571 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2395304AbfJPXG0 (ORCPT ); Wed, 16 Oct 2019 19:06:26 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Oct 2019 16:06:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,305,1566889200"; d="scan'208";a="189833638" Received: from unknown (HELO nsgsw-rhel7p6.lm.intel.com) ([10.232.116.93]) by orsmga008.jf.intel.com with ESMTP; 16 Oct 2019 16:06:25 -0700 From: Jon Derrick To: Lorenzo Pieralisi Cc: Bjorn Helgaas , , Pawel Baldysiak , Artur Paszkiewicz , Keith Busch , Dave Fugate , Andy Shevchenko , Jon Derrick Subject: [PATCH 3/3] PCI: vmd: Restore domain info during resets/unloads Date: Wed, 16 Oct 2019 11:04:48 -0600 Message-Id: <1571245488-3549-4-git-send-email-jonathan.derrick@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> References: <1571245488-3549-1-git-send-email-jonathan.derrick@intel.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Persistent domain info written by VMD BIOS needs to be restored by the driver during module unloads or resets. This adds a remove or reset action to restore the parsed domain info to all enabled VMD Root Ports. Signed-off-by: Jon Derrick Reviewed-by: Andy Shevchenko --- drivers/pci/controller/vmd.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c index dbe1bff..853aa93 100644 --- a/drivers/pci/controller/vmd.c +++ b/drivers/pci/controller/vmd.c @@ -562,6 +562,33 @@ static int vmd_pci_write(struct pci_bus *bus, unsigned int devfn, int reg, PCI_VENDOR_ID, 4, &temp) || \ temp == 0xffffffff) {} else +static void vmd_restore_domain(void *data) +{ + struct vmd_dev *vmd = data; + int root_port; + u32 temp, iobase; + + /* + * It shouldn't be possible for the Root Port layout to change + * dynamically (outside of BIOS), however there is no harm in writing + * the persistent data back to all enabled Root Ports. PCI resource + * assignment will discard any modifications on the next VMD domain bus + * scan following VMD reset/probe. + */ + for_each_vmd_root_port(vmd, root_port, temp) { + if (vmd_cfg_read(vmd, 0, PCI_DEVFN(root_port, 0), + PCI_IO_BASE, 2, &iobase)) + return; + + iobase &= ~((0xf << 4) | (0x3 << 14)); + iobase |= vmd->socket_nr << 4 | vmd->instance_nr << 14; + + if (vmd_cfg_write(vmd, 0, PCI_DEVFN(root_port, 0), + PCI_IO_BASE, 2, iobase)) + return; + } +} + static int vmd_parse_domain(struct vmd_dev *vmd) { int root_port, ret; @@ -579,6 +606,11 @@ static int vmd_parse_domain(struct vmd_dev *vmd) vmd->socket_nr = (iobase >> 4) & 0xf; vmd->instance_nr = (iobase >> 14) & 0x3; + ret = devm_add_action_or_reset(&vmd->dev->dev, + vmd_restore_domain, vmd); + if (ret) + return ret; + /* First available will be used */ break; }