From patchwork Wed Mar 6 22:11:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuppuswamy Sathyanarayanan X-Patchwork-Id: 10841865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 207CC1515 for ; Wed, 6 Mar 2019 22:14:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 068ED2EF59 for ; Wed, 6 Mar 2019 22:14:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA9E92EF5C; Wed, 6 Mar 2019 22:14:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85FBD2EF59 for ; Wed, 6 Mar 2019 22:14:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726264AbfCFWOh (ORCPT ); Wed, 6 Mar 2019 17:14:37 -0500 Received: from mga07.intel.com ([134.134.136.100]:13419 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726249AbfCFWOh (ORCPT ); Wed, 6 Mar 2019 17:14:37 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2019 14:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,449,1544515200"; d="scan'208";a="149937646" Received: from skuppusw-desk.jf.intel.com ([10.54.74.33]) by fmsmga004.fm.intel.com with ESMTP; 06 Mar 2019 14:13:33 -0800 From: sathyanarayanan.kuppuswamy@linux.intel.com To: bhelgaas@google.com, corbet@lwn.net Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Kuppuswamy Sathyanarayanan , Ashok Raj , Keith Busch Subject: [PATCH v1 1/5] PCI/IOV: Add support to verify PF/VF spec compliance Date: Wed, 6 Mar 2019 14:11:14 -0800 Message-Id: <317aaed09820db65c21452663bed33468f874d3a.1551909341.git.sathyanarayanan.kuppuswamy@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Kuppuswamy Sathyanarayanan PF/VF implementation must comply with PCIe specification as defined in r4.0, sec 9.3.4, 9.3.5, 9.3.6 and 9.3.7. And if it does not comply, return error and skip PF/VF device creation. Also add a command line parameter support to skip error when PF/VF spec validation failed. Cc: Ashok Raj Cc: Keith Busch Suggested-by: Ashok Raj Signed-off-by: Kuppuswamy Sathyanarayanan --- .../admin-guide/kernel-parameters.txt | 2 + drivers/pci/iov.c | 468 ++++++++++++++++++ drivers/pci/pci.c | 2 + drivers/pci/pci.h | 6 + include/linux/pci.h | 30 +- include/uapi/linux/pci_regs.h | 15 +- 6 files changed, 520 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 858b6c0b9a15..9e84b5f9c58d 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3363,6 +3363,8 @@ bridges without forcing it upstream. Note: this removes isolation between devices and may put more devices in an IOMMU group. + noiov_iverror Don't skip PCIe device enumeration, if VF/PF + function is not PCIe specification compliant. pcie_aspm= [PCIE] Forcibly enable or disable PCIe Active State Power Management. diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 3aa115ed3a65..9b121a649b90 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -17,6 +17,14 @@ #define VIRTFN_ID_LEN 16 +/* IOV invalid error */ +static int pci_iov_iverror = 1; + +void pci_noiov_iverror(void) +{ + pci_iov_iverror = 0; +} + int pci_iov_virtfn_bus(struct pci_dev *dev, int vf_id) { if (!dev->is_physfn) @@ -136,6 +144,455 @@ static void pci_read_vf_config_common(struct pci_dev *virtfn) physfn->sriov->cfg_size = pci_cfg_space_size(virtfn); } +static int pci_iov_physfn_valid(struct pci_dev *pdev) +{ + int status = 0, cap; + + if (!pdev->is_physfn) + return -EINVAL; + + /* + * Per PCIe r4.0, sec 9.3.7.9, PF must not implement MRIOV + * Capability. + */ + cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MRIOV); + if (cap) { + status = -EINVAL; + pdev->invalid_cap |= PCI_IOV_INVALID_MRIOV; + dev_warn(&pdev->dev, "%s: %s %s\n", "PF", "MRIOV Capability", + "must not be implemented"); + } + + return status; +} + +static int pci_iov_virtfn_valid(struct pci_dev *vdev) +{ + struct pci_dev *pdev = vdev->physfn; + u16 vreg16, preg16; + u32 vreg32, preg32; + u64 vreg64, preg64; + int status = 0, cap; + + if (!vdev->is_virtfn) + return -EINVAL; + + /* + * Per PCIe r4.0, sec 9.3.4.1.3, in Command register, I/O Space + * Enable, Memory Space Enable and Interrupt Disable bits should + * be tied to 0 for VFs. + */ + pci_read_config_word(vdev, PCI_COMMAND, &vreg16); + if (vreg16 & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY | + PCI_COMMAND_INTX_DISABLE)) { + dev_warn(&vdev->dev, "%s: %s\n", "VF", + "Non compilaint value in COMMAND register"); + status = -EINVAL; + } + + /* + * Per PCIe r4.0, sec 9.3.4.1.6, Class Code value should match + * between PF and VF. + */ + pci_read_config_dword(vdev, PCI_CLASS_REVISION, &vreg32); + pci_read_config_dword(pdev, PCI_CLASS_REVISION, &preg32); + vreg32 = vreg32 >> 8; + preg32 = preg32 >> 8; + if (vreg32 != preg32) { + dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF", + "Class Code mismatch", vreg32, preg32); + status = -EINVAL; + } + + /* + * Per PCIe r4.0, sec 9.3.4.1.13, Subsystem Vendor ID value should + * match between PF and VF. + */ + pci_read_config_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, &vreg16); + pci_read_config_word(pdev, PCI_SUBSYSTEM_VENDOR_ID, &preg16); + if (vreg16 != preg16) { + dev_warn(&vdev->dev, "%s: %s %x!=%x\n", "PF/VF", + "Subsystem Vendor ID mismatch", vreg16, preg16); + status = -EINVAL; + } + + /* + * Per PCIe r4.0, sec 9.3.6, VF must not implement Enhanced Allocation + * Capability. + */ + cap = pci_find_capability(vdev, PCI_CAP_ID_EA); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_EA; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Enhanced Allocation Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.6, VF must not implement Native PCIe + * Enclosure Management (NPEM) Capability. + */ + cap = pci_find_capability(vdev, PCI_CAP_ID_NPEM); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_NPEM; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Native PCIe Native PCIe Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_VC; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Virtual Channel Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Multi Function + * Virtual Channel Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MFVC); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_MFVC; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Multi Function VC Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.1, VF must not implement Virtual Channel(9) + * Virtual Channel Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VC9); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_VC9; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Virtual Channel(9) Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.2, VF can optionally implement Device + * Serial Number Capability. But if it implements it, the value + * should match the PF. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DSN); + if (cap) { + pci_read_config_dword(vdev, cap + PCI_DSN_SNUR, &vreg32); + vreg64 = (u64)vreg32 << 32; + pci_read_config_dword(vdev, cap + PCI_DSN_SNLR, &vreg32); + vreg64 |= vreg32; + cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN); + if (cap) { + pci_read_config_dword(pdev, cap + PCI_DSN_SNUR, + &preg32); + preg64 = (u64)preg32 << 32; + pci_read_config_dword(pdev, cap + PCI_DSN_SNLR, + &preg32); + preg64 |= preg32; + } + if (!cap || vreg64 != preg64) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_DSN; + dev_warn(&vdev->dev, "%s: %s\n", "PF/VF", + "Device Serial Number mismatch"); + } + } + + /* + * Per PCIe r4.0, sec 9.3.7.3, VF must not implement Power Budgeting + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PWR); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PWR; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Power Budgeting Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.8, if VF implements Address Translation + * Services (ATS) Extended Capability then corresponding PF should + * also implement it. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_ATS); + if (cap) { + cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ATS); + if (!cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_ATS; + dev_warn(&vdev->dev, "%s: %s\n", "PF/VF", + "ATS Capability mismatch"); + } + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement SRIOV + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SRIOV); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_SRIOV; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "SRIOV Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.9, VF must not implement MRIOV + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MRIOV); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_MRIOV; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "MRIOV Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.10, if VF implements Multicast + * Capability then corresponding PF should also implement it. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MCAST); + if (cap) { + cap = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_MCAST); + if (!cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_MCAST; + dev_warn(&vdev->dev, "%s: %s\n", "PF/VF", + "Multicast Capability mismatch"); + } + } + + /* + * Per PCIe r4.0, sec 9.3.7.11, VF must not implement PRI + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PRI); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PRI; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", "PRI Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.4, VF must not implement Resizable BAR + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_REBAR); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_REBAR; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Resizable BAR Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.4, VF must not implement VF Resizable BAR + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_VREBAR); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_VREBAR; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "VF Resizable BAR Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.12, VF must not implement Dynamic Power + * Allocation Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DPA); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_DPA; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Dynamic Power Allocation Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Latency Tolerance + * Reporting (LTR) Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_LTR); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_LTR; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Latency Tolerance Reporting Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Secondary PCIe + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_SECPCI); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_SECPCI; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Secondary PCIe Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Protocol + * Multiplexing Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PMUX); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PMUX; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Protocol Multiplexing Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7.14, VF must not implement Process Address + * Space ID (PASID) Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PASID); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PASID; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Process Address Space ID Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement L1 PM Substates + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_L1SS); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_L1SS; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "L1 PM Substates Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Precision Time + * Measurement Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PTM); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PTM; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Precision Time Measurement", + "Capability must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement PCI Express + * over M-PHY Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_MPHY); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_MPHY; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "PCI Express over M-PHY Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Data Link Feature + * Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_DLF); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_DLF; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Data Link Feature Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0 + * GT/s Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PL16; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Physical Layer 16.0 GT/s Capability", + "must not be implemented"); + } + + /* + * Per PCIe r4.0, sec 9.3.7, VF must not implement Physical Layer 16.0 + * GT/s Margining Capability. + */ + cap = pci_find_ext_capability(vdev, PCI_EXT_CAP_ID_PL16M); + if (cap) { + status = -EINVAL; + vdev->invalid_cap |= PCI_IOV_INVALID_PL16M; + dev_warn(&vdev->dev, "%s: %s %s\n", "VF", + "Physical Layer 16.0 GT/s Margining Capability", + "must not be implemented"); + } + + if (vdev->pcie_cap) { + /* + * Per PCIe r4.0, sec 9.3.5.3, in VF device, Device + * Capabilities Register Phantom Functions Supported + * bit should be tied to 0 and Function Level Reset + * Capability bit should be tied to 1. + */ + pcie_capability_read_dword(vdev, PCI_EXP_DEVCAP, &vreg32); + if (vreg32 & PCI_EXP_DEVCAP_PHANTOM) { + dev_warn(&vdev->dev, "%s: %s\n", "VF", + "Phantom Functions Supported bit is invalid"); + status = -EINVAL; + } + if (!(vreg32 & PCI_EXP_DEVCAP_FLR)) { + dev_warn(&vdev->dev, "%s: %s\n", "VF", + "Function Level Reset bit is invalid"); + status = -EINVAL; + } + + /* + * Per PCIe r4.0, sec 9.3.5.5, in VF device, Device Status + * Register AUX Power Detected bit and Emergency Power + * Reduction Detected bits should be tied to 0. + */ + pcie_capability_read_word(vdev, PCI_EXP_DEVSTA, &vreg16); + if (vreg16 & (PCI_EXP_DEVSTA_AUXPD | PCI_EXP_DEVSTA_EPRD)) { + dev_warn(&vdev->dev, "%s: %s\n", "VF", + "Device Status register value is invalid"); + status = -EINVAL; + } + } + + return status; +} + int pci_iov_add_virtfn(struct pci_dev *dev, int id) { int i; @@ -186,6 +643,11 @@ int pci_iov_add_virtfn(struct pci_dev *dev, int id) pci_device_add(virtfn, virtfn->bus); + /* Verify whether VF complies with spec */ + rc = pci_iov_virtfn_valid(virtfn); + if (rc < 0 && pci_iov_iverror) + goto failed2; + sprintf(buf, "virtfn%u", id); rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf); if (rc) @@ -511,6 +973,12 @@ static int sriov_init(struct pci_dev *dev, int pos) dev->sriov = iov; dev->is_physfn = 1; + + /* Verify whether PF complies with spec */ + rc = pci_iov_physfn_valid(dev); + if (rc < 0 && pci_iov_iverror) + goto fail_max_buses; + rc = compute_max_vf_buses(dev); if (rc) goto fail_max_buses; diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c25acace7d91..67fac64ba112 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -6197,6 +6197,8 @@ static int __init pci_setup(char *str) } else if (!strncmp(str, "disable_acs_redir=", 18)) { disable_acs_redir_param = kstrdup(str + 18, GFP_KERNEL); + } else if (!strcmp(str, "noiov_iverror")) { + pci_noiov_iverror(); } else { printk(KERN_ERR "PCI: Unknown option `%s'\n", str); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 224d88634115..9cc04271b715 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -602,4 +602,10 @@ static inline void pci_aer_clear_fatal_status(struct pci_dev *dev) { } static inline void pci_aer_clear_device_status(struct pci_dev *dev) { } #endif +#ifdef CONFIG_PCI_IOV +void pci_noiov_iverror(void); +#else +static inline void pci_noiov_iverror(void) { } +#endif + #endif /* DRIVERS_PCI_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index 65f1d8c2f082..489fc0f68bb1 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -435,6 +435,7 @@ struct pci_dev { #ifdef CONFIG_PCI_MSI const struct attribute_group **msi_irq_groups; #endif + u64 invalid_cap; /* PF/VF: Invalid Capability bitmap*/ struct pci_vpd *vpd; #ifdef CONFIG_PCI_ATS union { @@ -446,7 +447,7 @@ struct pci_dev { atomic_t ats_ref_cnt; /* Number of VFs with ATS enabled */ #endif #ifdef CONFIG_PCI_PRI - u32 pri_reqs_alloc; /* Number of PRI requests allocated */ + u32 pri_reqs_alloc; /* Number of PRI requests allocated */ #endif #ifdef CONFIG_PCI_PASID u16 pasid_features; @@ -1953,6 +1954,33 @@ extern int pci_pci_problems; #define PCIPCI_ALIMAGIK 32 /* Need low latency setting */ #define PCIAGP_FAIL 64 /* No PCI to AGP DMA */ +/* invalid_cap bitmap definitions*/ +#define PCI_IOV_INVALID_EA BIT(0) /* Invalid EA Capability */ +#define PCI_IOV_INVALID_NPEM BIT(1) /* Invalid NPEM Capability */ +#define PCI_IOV_INVALID_VC BIT(2) /* Invalid VC Capability */ +#define PCI_IOV_INVALID_MFVC BIT(3) /* Invalid MFVC Capability */ +#define PCI_IOV_INVALID_VC9 BIT(4) /* Invalid VC9 Capability */ +#define PCI_IOV_INVALID_DSN BIT(5) /* Invalid DSN Capability */ +#define PCI_IOV_INVALID_PWR BIT(6) /* Invalid PWR Capability */ +#define PCI_IOV_INVALID_ATS BIT(7) /* Invalid ATS Capability */ +#define PCI_IOV_INVALID_SRIOV BIT(8) /* Invalid SRIOV Capability */ +#define PCI_IOV_INVALID_MRIOV BIT(9) /* Invalid MRIOV Capability */ +#define PCI_IOV_INVALID_MCAST BIT(10) /* Invalid MCAST Capability */ +#define PCI_IOV_INVALID_PRI BIT(11) /* Invalid PRI Capability */ +#define PCI_IOV_INVALID_REBAR BIT(12) /* Invalid REBAR Capability */ +#define PCI_IOV_INVALID_VREBAR BIT(13) /* Invalid VREBAR Capability */ +#define PCI_IOV_INVALID_DPA BIT(14) /* Invalid DPA Capability */ +#define PCI_IOV_INVALID_LTR BIT(15) /* Invalid LTR Capability */ +#define PCI_IOV_INVALID_SECPCI BIT(16) /* Invalid SECPCI Capability */ +#define PCI_IOV_INVALID_PMUX BIT(17) /* Invalid PMUX Capability */ +#define PCI_IOV_INVALID_PASID BIT(18) /* Invalid PASID Capability */ +#define PCI_IOV_INVALID_L1SS BIT(19) /* Invalid L1SS Capability */ +#define PCI_IOV_INVALID_PTM BIT(20) /* Invalid PTM Capability */ +#define PCI_IOV_INVALID_MPHY BIT(21) /* Invalid MPHY Capability */ +#define PCI_IOV_INVALID_DLF BIT(22) /* Invalid DLF Capability */ +#define PCI_IOV_INVALID_PL16 BIT(23) /* Invalid PL16 Capability */ +#define PCI_IOV_INVALID_PL16M BIT(24) /* Invalid PL16M Capability */ + extern unsigned long pci_cardbus_io_size; extern unsigned long pci_cardbus_mem_size; extern u8 pci_dfl_cache_line_size; diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index e1e9888c85e6..d377d48ee99c 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -227,7 +227,8 @@ #define PCI_CAP_ID_SATA 0x12 /* SATA Data/Index Conf. */ #define PCI_CAP_ID_AF 0x13 /* PCI Advanced Features */ #define PCI_CAP_ID_EA 0x14 /* PCI Enhanced Allocation */ -#define PCI_CAP_ID_MAX PCI_CAP_ID_EA +#define PCI_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */ +#define PCI_CAP_ID_MAX PCI_CAP_ID_NPEM #define PCI_CAP_LIST_NEXT 1 /* Next capability in the list */ #define PCI_CAP_FLAGS 2 /* Capability defined flags (16 bits) */ #define PCI_CAP_SIZEOF 4 @@ -517,6 +518,7 @@ #define PCI_EXP_DEVSTA_URD 0x0008 /* Unsupported Request Detected */ #define PCI_EXP_DEVSTA_AUXPD 0x0010 /* AUX Power Detected */ #define PCI_EXP_DEVSTA_TRPND 0x0020 /* Transactions Pending */ +#define PCI_EXP_DEVSTA_EPRD 0x0040 /* Emergency Power Reduction Detected */ #define PCI_CAP_EXP_RC_ENDPOINT_SIZEOF_V1 12 /* v1 endpoints without link end here */ #define PCI_EXP_LNKCAP 12 /* Link Capabilities */ #define PCI_EXP_LNKCAP_SLS 0x0000000f /* Supported Link Speeds */ @@ -705,7 +707,12 @@ #define PCI_EXT_CAP_ID_DPC 0x1D /* Downstream Port Containment */ #define PCI_EXT_CAP_ID_L1SS 0x1E /* L1 PM Substates */ #define PCI_EXT_CAP_ID_PTM 0x1F /* Precision Time Measurement */ -#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PTM +#define PCI_EXT_CAP_ID_MPHY 0x20 /* PCI Express over M-PHY (M-PCIe) */ +#define PCI_EXT_CAP_ID_VREBAR 0x24 /* VF Resizable BAR */ +#define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ +#define PCI_EXT_CAP_ID_PL16 0x26 /* Physical Layer 16.0 GT/s */ +#define PCI_EXT_CAP_ID_PL16M 0x27 /* Physical Layer 16.0 GT/s Margining*/ +#define PCI_EXT_CAP_ID_MAX PCI_EXT_CAP_ID_PL16M #define PCI_EXT_CAP_DSN_SIZEOF 12 #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40 @@ -798,6 +805,10 @@ #define PCI_CAP_VC_BASE_SIZEOF 0x10 #define PCI_CAP_VC_PER_VC_SIZEOF 0x0C +/* Device Serial Number */ +#define PCI_DSN_SNLR 0x04 /* Serial Number Lower Register */ +#define PCI_DSN_SNUR 0x08 /* Serial Number Upper Register */ + /* Power Budgeting */ #define PCI_PWR_DSR 4 /* Data Select Register */ #define PCI_PWR_DATA 8 /* Data Register */ From patchwork Wed Mar 6 22:11:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuppuswamy Sathyanarayanan X-Patchwork-Id: 10841863 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFA9D1515 for ; Wed, 6 Mar 2019 22:13:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DBF631FFE5 for ; Wed, 6 Mar 2019 22:13:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE18F2E39B; Wed, 6 Mar 2019 22:13:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 617D21FFE5 for ; Wed, 6 Mar 2019 22:13:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726738AbfCFWNg (ORCPT ); Wed, 6 Mar 2019 17:13:36 -0500 Received: from mga07.intel.com ([134.134.136.100]:13420 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726364AbfCFWNg (ORCPT ); Wed, 6 Mar 2019 17:13:36 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2019 14:13:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,449,1544515200"; d="scan'208";a="149937650" Received: from skuppusw-desk.jf.intel.com ([10.54.74.33]) by fmsmga004.fm.intel.com with ESMTP; 06 Mar 2019 14:13:34 -0800 From: sathyanarayanan.kuppuswamy@linux.intel.com To: bhelgaas@google.com, corbet@lwn.net Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Kuppuswamy Sathyanarayanan , Ashok Raj , Keith Busch Subject: [PATCH v1 2/5] PCI/ATS: Fix PRI PF/VF dependency issues Date: Wed, 6 Mar 2019 14:11:15 -0800 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Kuppuswamy Sathyanarayanan As per PCIe spec r4.0, sec 9.3.7.11 ("Page Request Interface (PRI)"), all VFs associated with PF can only use the Page Request Interface of the PF and not implement it. So for any PRI capability related queries on a VF device use associated PF device capabilities. Also disable PRI on PF only when all related VFs disable PRI. Cc: Ashok Raj Cc: Keith Busch Suggested-by: Ashok Raj Signed-off-by: Kuppuswamy Sathyanarayanan --- drivers/pci/ats.c | 47 ++++++++++++++++++++++++++++++++++++++++++++- include/linux/pci.h | 1 + 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 5b78f3b1b918..3fcef4544c4c 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -154,10 +154,33 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs) u16 control, status; u32 max_requests; int pos; + struct pci_dev *pf; if (WARN_ON(pdev->pri_enabled)) return -EBUSY; + /* If PRI Capability is invalid, return error */ + if (pdev->is_virtfn || pdev->is_physfn) { + if (pdev->invalid_cap & PCI_IOV_INVALID_PRI) + return -EINVAL; + } + + if (pdev->is_virtfn) { + pf = pci_physfn(pdev); + + /* If VF config does not match with PF, return error */ + if (!pf->pri_enabled) + return -EINVAL; + + pdev->pri_reqs_alloc = pf->pri_reqs_alloc; + pdev->pri_enabled = 1; + + /* Increment PF PRI refcount */ + atomic_inc(&pf->pri_ref_cnt); + + return 0; + } + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI); if (!pos) return -EINVAL; @@ -175,7 +198,6 @@ int pci_enable_pri(struct pci_dev *pdev, u32 reqs) pci_write_config_word(pdev, pos + PCI_PRI_CTRL, control); pdev->pri_enabled = 1; - return 0; } EXPORT_SYMBOL_GPL(pci_enable_pri); @@ -190,10 +212,27 @@ void pci_disable_pri(struct pci_dev *pdev) { u16 control; int pos; + struct pci_dev *pf; if (WARN_ON(!pdev->pri_enabled)) return; + /* All VFs should be disabled before disabling PF */ + if (atomic_read(&pdev->pri_ref_cnt)) + return; + + if (pdev->is_virtfn) { + /* Since VF shares PRI with PF, use PF config. */ + pf = pci_physfn(pdev); + + /* Decrement PF PRI refcount */ + atomic_dec(&pf->pri_ref_cnt); + + pdev->pri_enabled = 0; + + return; + } + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI); if (!pos) return; @@ -219,6 +258,9 @@ void pci_restore_pri_state(struct pci_dev *pdev) if (!pdev->pri_enabled) return; + if (pdev->is_virtfn) + return; + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI); if (!pos) return; @@ -243,6 +285,9 @@ int pci_reset_pri(struct pci_dev *pdev) if (WARN_ON(pdev->pri_enabled)) return -EBUSY; + if (pdev->is_virtfn) + return 0; + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI); if (!pos) return -EINVAL; diff --git a/include/linux/pci.h b/include/linux/pci.h index 489fc0f68bb1..d5df80ab2645 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -448,6 +448,7 @@ struct pci_dev { #endif #ifdef CONFIG_PCI_PRI u32 pri_reqs_alloc; /* Number of PRI requests allocated */ + atomic_t pri_ref_cnt; /* Number of VFs with PRI enabled */ #endif #ifdef CONFIG_PCI_PASID u16 pasid_features; From patchwork Wed Mar 6 22:11:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuppuswamy Sathyanarayanan X-Patchwork-Id: 10841859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00DF41575 for ; Wed, 6 Mar 2019 22:13:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF13B2E0C4 for ; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D11B52EF4F; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E1F82EF50 for ; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726768AbfCFWNi (ORCPT ); Wed, 6 Mar 2019 17:13:38 -0500 Received: from mga07.intel.com ([134.134.136.100]:13420 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726388AbfCFWNh (ORCPT ); Wed, 6 Mar 2019 17:13:37 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2019 14:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,449,1544515200"; d="scan'208";a="149937654" Received: from skuppusw-desk.jf.intel.com ([10.54.74.33]) by fmsmga004.fm.intel.com with ESMTP; 06 Mar 2019 14:13:34 -0800 From: sathyanarayanan.kuppuswamy@linux.intel.com To: bhelgaas@google.com, corbet@lwn.net Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Kuppuswamy Sathyanarayanan , Ashok Raj , Keith Busch Subject: [PATCH v1 3/5] PCI/ATS: Fix PASID PF/VF dependency issues Date: Wed, 6 Mar 2019 14:11:16 -0800 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Kuppuswamy Sathyanarayanan As per PCIe spec r4.0, sec 9.3.7.14 ("PASID"), all VFs associated with PF can only use the PASID of the PF and not implement it. So for any PASID capability related queries on a VF device use associated PF device capabilities. Also disable PASID support on a PF device only when all related VFs disable PASID. Cc: Ashok Raj Cc: Keith Busch Suggested-by: Ashok Raj Signed-off-by: Kuppuswamy Sathyanarayanan --- drivers/pci/ats.c | 62 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/pci.h | 1 + 2 files changed, 63 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 3fcef4544c4c..28bbf7dad425 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -314,6 +314,7 @@ int pci_enable_pasid(struct pci_dev *pdev, int features) { u16 control, supported; int pos; + struct pci_dev *pf; if (WARN_ON(pdev->pasid_enabled)) return -EBUSY; @@ -321,6 +322,29 @@ int pci_enable_pasid(struct pci_dev *pdev, int features) if (!pdev->eetlp_prefix_path) return -EINVAL; + /* If PASID capability is invalid, return error */ + if (pdev->is_virtfn || pdev->is_physfn) { + if (pdev->invalid_cap & PCI_IOV_INVALID_PASID) + return -EINVAL; + } + + if (pdev->is_virtfn) { + /* Since VF shares PASID with PF, use PF config */ + pf = pci_physfn(pdev); + + /* If VF config does not match with PF, return error */ + if (!pf->pasid_enabled || pf->pasid_features != features) + return -EINVAL; + + pdev->pasid_features = features; + pdev->pasid_enabled = 1; + + /* Increment PF PASID refcount */ + atomic_inc(&pf->pasid_ref_cnt); + + return 0; + } + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID); if (!pos) return -EINVAL; @@ -351,10 +375,27 @@ void pci_disable_pasid(struct pci_dev *pdev) { u16 control = 0; int pos; + struct pci_dev *pf; if (WARN_ON(!pdev->pasid_enabled)) return; + /* All VFs PASID should be disabled before disabling PF PASID*/ + if (atomic_read(&pdev->pasid_ref_cnt)) + return; + + if (pdev->is_virtfn) { + /* Since VF shares PASID with PF, use PF config. */ + pf = pci_physfn(pdev); + + /* Decrement PF PASID refcount */ + atomic_dec(&pf->pasid_ref_cnt); + + pdev->pasid_enabled = 0; + + return; + } + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID); if (!pos) return; @@ -377,6 +418,9 @@ void pci_restore_pasid_state(struct pci_dev *pdev) if (!pdev->pasid_enabled) return; + if (pdev->is_virtfn) + return; + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID); if (!pos) return; @@ -401,6 +445,15 @@ int pci_pasid_features(struct pci_dev *pdev) u16 supported; int pos; + /* If PASID Cap is invalid then return error */ + if (pdev->is_virtfn || pdev->is_physfn) { + if (pdev->invalid_cap & PCI_IOV_INVALID_PASID) + return -EINVAL; + } + + if (pdev->is_virtfn) + pdev = pci_physfn(pdev); + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID); if (!pos) return -EINVAL; @@ -427,6 +480,15 @@ int pci_max_pasids(struct pci_dev *pdev) u16 supported; int pos; + /* If PASID Cap is invalid then return error */ + if (pdev->is_virtfn || pdev->is_physfn) { + if (pdev->invalid_cap & PCI_IOV_INVALID_PASID) + return -EINVAL; + } + + if (pdev->is_virtfn) + pdev = pci_physfn(pdev); + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PASID); if (!pos) return -EINVAL; diff --git a/include/linux/pci.h b/include/linux/pci.h index d5df80ab2645..c6c413c52403 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -452,6 +452,7 @@ struct pci_dev { #endif #ifdef CONFIG_PCI_PASID u16 pasid_features; + atomic_t pasid_ref_cnt; /* Number of VFs with PASID enabled */ #endif #ifdef CONFIG_PCI_P2PDMA struct pci_p2pdma *p2pdma; From patchwork Wed Mar 6 22:11:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuppuswamy Sathyanarayanan X-Patchwork-Id: 10841857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5AC9A1515 for ; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 467922E0C4 for ; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 372D52EF4F; Wed, 6 Mar 2019 22:13:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAF3F2E0C4 for ; Wed, 6 Mar 2019 22:13:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726774AbfCFWNi (ORCPT ); Wed, 6 Mar 2019 17:13:38 -0500 Received: from mga07.intel.com ([134.134.136.100]:13420 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726364AbfCFWNh (ORCPT ); Wed, 6 Mar 2019 17:13:37 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2019 14:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,449,1544515200"; d="scan'208";a="149937658" Received: from skuppusw-desk.jf.intel.com ([10.54.74.33]) by fmsmga004.fm.intel.com with ESMTP; 06 Mar 2019 14:13:35 -0800 From: sathyanarayanan.kuppuswamy@linux.intel.com To: bhelgaas@google.com, corbet@lwn.net Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Kuppuswamy Sathyanarayanan , Ashok Raj , Keith Busch Subject: [PATCH v1 4/5] PCI/ATS: For PF/VF skip ATS initalization if spec check failed Date: Wed, 6 Mar 2019 14:11:17 -0800 Message-Id: <0898a9bd92cfcf2ab07e5dd5ba59dc181b88edec.1551909341.git.sathyanarayanan.kuppuswamy@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Kuppuswamy Sathyanarayanan Don't initialize ATS for VF/PF devices if the ATS Capability implementaion is not aligned with PCIe spec r4.0 sec 9.3.7.8. Cc: Ashok Raj Cc: Keith Busch Suggested-by: Ashok Raj Reviewed-by: Keith Busch Signed-off-by: Kuppuswamy Sathyanarayanan --- drivers/pci/ats.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 28bbf7dad425..11299d93a59a 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -23,6 +23,12 @@ void pci_ats_init(struct pci_dev *dev) if (pci_ats_disabled()) return; + /* If ATS Cap is invalid then return */ + if (dev->is_virtfn || dev->is_physfn) { + if (dev->invalid_cap & PCI_IOV_INVALID_ATS) + return; + } + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ATS); if (!pos) return; From patchwork Wed Mar 6 22:11:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuppuswamy Sathyanarayanan X-Patchwork-Id: 10841861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFE121515 for ; Wed, 6 Mar 2019 22:13:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C83A2EF4F for ; Wed, 6 Mar 2019 22:13:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9119A2EF59; Wed, 6 Mar 2019 22:13:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2724C2EF4F for ; Wed, 6 Mar 2019 22:13:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726806AbfCFWNs (ORCPT ); Wed, 6 Mar 2019 17:13:48 -0500 Received: from mga07.intel.com ([134.134.136.100]:13421 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726763AbfCFWNh (ORCPT ); Wed, 6 Mar 2019 17:13:37 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Mar 2019 14:13:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,449,1544515200"; d="scan'208";a="149937662" Received: from skuppusw-desk.jf.intel.com ([10.54.74.33]) by fmsmga004.fm.intel.com with ESMTP; 06 Mar 2019 14:13:35 -0800 From: sathyanarayanan.kuppuswamy@linux.intel.com To: bhelgaas@google.com, corbet@lwn.net Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, Kuppuswamy Sathyanarayanan , Ashok Raj , Keith Busch Subject: [PATCH v1 5/5] PCI/ATS: Fix ATS PF/VF dependency issues Date: Wed, 6 Mar 2019 14:11:18 -0800 Message-Id: <770df406d036606f1f157a7df0d462d11ee5fae0.1551909341.git.sathyanarayanan.kuppuswamy@linux.intel.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Kuppuswamy Sathyanarayanan As per PCIe spec r4.0, sec 9.3.7.8, ATS Capabilities in VFs and their associated PFs may be enabled independently. But currently all VFs needs to disable ATS service before disabling the ATS service in PF. So remove this dependency logic in enable/disable code. Cc: Ashok Raj Cc: Keith Busch Suggested-by: Ashok Raj Signed-off-by: Kuppuswamy Sathyanarayanan --- drivers/pci/ats.c | 11 ----------- include/linux/pci.h | 1 - 2 files changed, 12 deletions(-) diff --git a/drivers/pci/ats.c b/drivers/pci/ats.c index 11299d93a59a..062471abfeca 100644 --- a/drivers/pci/ats.c +++ b/drivers/pci/ats.c @@ -66,8 +66,6 @@ int pci_enable_ats(struct pci_dev *dev, int ps) pdev = pci_physfn(dev); if (pdev->ats_stu != ps) return -EINVAL; - - atomic_inc(&pdev->ats_ref_cnt); /* count enabled VFs */ } else { dev->ats_stu = ps; ctrl |= PCI_ATS_CTRL_STU(dev->ats_stu - PCI_ATS_MIN_STU); @@ -85,20 +83,11 @@ EXPORT_SYMBOL_GPL(pci_enable_ats); */ void pci_disable_ats(struct pci_dev *dev) { - struct pci_dev *pdev; u16 ctrl; if (WARN_ON(!dev->ats_enabled)) return; - if (atomic_read(&dev->ats_ref_cnt)) - return; /* VFs still enabled */ - - if (dev->is_virtfn) { - pdev = pci_physfn(dev); - atomic_dec(&pdev->ats_ref_cnt); - } - pci_read_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, &ctrl); ctrl &= ~PCI_ATS_CTRL_ENABLE; pci_write_config_word(dev, dev->ats_cap + PCI_ATS_CTRL, ctrl); diff --git a/include/linux/pci.h b/include/linux/pci.h index c6c413c52403..07e796e7f2bf 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -444,7 +444,6 @@ struct pci_dev { }; u16 ats_cap; /* ATS Capability offset */ u8 ats_stu; /* ATS Smallest Translation Unit */ - atomic_t ats_ref_cnt; /* Number of VFs with ATS enabled */ #endif #ifdef CONFIG_PCI_PRI u32 pri_reqs_alloc; /* Number of PRI requests allocated */