From patchwork Fri Oct 27 02:37:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stephen Bates X-Patchwork-Id: 10028993 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 21EF160375 for ; Fri, 27 Oct 2017 02:38:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 115DB28E42 for ; Fri, 27 Oct 2017 02:38:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 051EB28ED5; Fri, 27 Oct 2017 02:38:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 12EFE28E42 for ; Fri, 27 Oct 2017 02:38:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751520AbdJ0CiC (ORCPT ); Thu, 26 Oct 2017 22:38:02 -0400 Received: from mail-sn1nam02on0092.outbound.protection.outlook.com ([104.47.36.92]:21669 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751457AbdJ0CiA (ORCPT ); Thu, 26 Oct 2017 22:38:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raithlin.onmicrosoft.com; s=selector1-raithlin-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=wAyR8mj1zMghXfwzfbvcPST1Tio8CA448c8pQ+UXAJs=; b=ZKQGgdPNwtP4cCkGZLzvOBqzk8uBMbdOwg/vUio3JKM7bzwfnD7LnRFF5mvq+Wjnvbv0Cy24lnqJU0MWUrqygyKi+29jw7OuyZWfHMDh4eskfthvke3QUwhSr4/0oSSr9KiBbj5iv18glCq7tDtspAkUK4iU393aembfqKanCns= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=sbates@raithlin.com; Received: from ubuntu64-batesste.cn.shawcable.net (70.65.224.121) by YTXPR0101MB2046.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:10::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Fri, 27 Oct 2017 02:37:55 +0000 From: sbates@raithlin.com To: corbet@lwn.net, bhelgaas@google.com, paulmck@linux.vnet.ibm.com, tglx@linutronix.de, akpm@linux-foundation.org, mingo@kernel.org, cdall@linaro.org, marc.zyngier@arm.com, zohar@linux.vnet.ibm.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org Cc: logang@deltatee.com, tom@talpey.com, Stephen Bates Subject: [PATCH] pci: Add a acs_disable option for pci kernel parameter Date: Thu, 26 Oct 2017 20:37:49 -0600 Message-Id: <1509071869-18771-1-git-send-email-sbates@raithlin.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [70.65.224.121] X-ClientProxiedBy: CY4PR08CA0044.namprd08.prod.outlook.com (2603:10b6:903:151::30) To YTXPR0101MB2046.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:10::10) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 296b766f-b423-403d-c5fe-08d51ce3bb9a X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4603075)(4627075)(201702281549075)(2017052603199); SRVR:YTXPR0101MB2046; X-Microsoft-Exchange-Diagnostics: 1; YTXPR0101MB2046; 3:L/2MnodO1bSYejXczsDZvUg4mP/0QqP0RxZWX6mHh6jSQ16pCozHRW7vHwNH9X+4cuKIIndSNi6y+ghVmgu3xK0aCgYn5dcbkfXOnq3nH4BG0Sh+87ueE7HQ7rSCWxMEDOhULpCGPXMTthn17XofscQS0TSvaq8C5vjarjlSQyNXXhPi/0QEk+pTBl9zKdv+CvOpK1NjNLn9/t9Z9op8LTA6ObrdjayDELhkJuaXMvGbJtGn6sDk80ATTHVBsU5L; 25:K3buBABDxfuTakAeNTEBhrqXOPLw99zF042I8sdEl5zu+E7y58lbZf5Fu7ZmUWNf1nDfB8FltsayVlIArqpPbk5/8Rlf66sxkK95C16K07wlLDkaYKuu+mdAAZsUF705oEhuMtESVVgNmBcONR460qdkmCgl3qoMalL9+Qa76VGtsvEmdfuMGcehAh8m8vDvMrG27MTbgGVGZoyYbP6qXazEoym+pclZLgvg1HQ24yE5XL8AtIGsCxAbQzwHOQJiu6+bLW7wvgAh+YgqN+jmLY3juDa7mb05rVQVMrT3/HSgzfg/6vp6gjd5c0IVuQNPP/eZCyvDEvITecodEsCS1A==; 31:UC0R+pAnHdZyeoaDZ08PjBQrQkp/ZycMpo3l0grcZ0Mvk1lBqOI35ruOLsUQ2fbRbi932GNIxQFqnm7K7mBsynIFPZk+e3KO6P98DBHqJG7NMSLZWxCfUp1zhYaDoqwVM+O1fY3LrVbQ/3JRlrYK0oMW3TMg85edGJ0OejhKt//ZzIeOv1JVOcowXY/+Myn4OqLNzvapePgiZFzJjSpfmlmlgMdQ7g7+I4N9/+PZOPY= X-MS-TrafficTypeDiagnostic: YTXPR0101MB2046: X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(93006095)(93001095)(3231020)(10201501046)(3002001)(6041248)(2016111802025)(20161123560025)(20161123555025)(20161123562025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(6072148)(6043046)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR0101MB2046; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR0101MB2046; X-Microsoft-Exchange-Diagnostics: 1; YTXPR0101MB2046; 4:TaRoWmlPBT7bxKsQOCsuOcNdi+5a0+Wk0XQeG6TK8hXR4qm+pxWeSwxjKHs9wNUz/aHEArBXEqY5u0csGNQh5j3ySR2gc8gaYUJn0CiuBcWYr1Lo09p1z9KvqupXqcXw6D+9zgItcOp8ugsGqsNbjKePDx/winXLNqKQQluD1+KGCqmGIRxwLHRJ9N11Kfu9Ni52eEGS1cStJHrkfemcef0L77CFz74wqEGsaYLMzB3Rx0OSc+DfiO8kdzs0Qt5AeBg0xt14IRv3mSOSM5U3Cg== X-Forefront-PRVS: 0473A03F3F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(6009001)(376002)(39830400002)(346002)(199003)(189002)(107886003)(97736004)(316002)(6486002)(478600001)(36756003)(16586007)(86362001)(9686003)(16526018)(6116002)(3846002)(53936002)(4326008)(25786009)(85782001)(50226002)(81166006)(105586002)(81156014)(33646002)(7736002)(189998001)(5003940100001)(6666003)(50466002)(2906002)(305945005)(7416002)(47776003)(6512007)(68736007)(8676002)(50986999)(101416001)(8936002)(66066001)(5660300001)(106356001)(48376002)(921003)(1121003)(85772001); DIR:OUT; SFP:1102; SCL:1; SRVR:YTXPR0101MB2046; H:ubuntu64-batesste.cn.shawcable.net; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: raithlin.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; YTXPR0101MB2046; 23:QQWmFysQKcyiTQsvX+dG3qgVOVTWGzc1LCpfVKb?= =?us-ascii?Q?ccfG+L40ytRKInpdBz2q2v2n/wAuWpY9dqkz0dHroKujgruxW3Kl+d6HpcBH?= =?us-ascii?Q?ZjxJFESnI8pM/E+QsEBqdRB//0UBsVY0m0P2cIzbbfrzL6vDt9cQF1ONuuCI?= =?us-ascii?Q?3RrnM+ZeML0roD2p91QDECC8ket0W2/yQVNHl/q4ykrs2M5GodHa57YYaEjO?= =?us-ascii?Q?iRS5q0siGVE/gpoLoK4vfKD2/qjimv67FJmytJar1sHCzX8KwtDdKYoGJmPl?= =?us-ascii?Q?t5Ur9sbxoyaXQd6XHsBZ6/fWambtvINxlf05Zdrn9r8vONcfN/X2Zs7P6+us?= =?us-ascii?Q?Y8+spK7ZYEBj3LSDn2zvqJzAkt/DnI+LGgHf1lHUO6Oq1MHr9ldeI9xCYaXU?= =?us-ascii?Q?TjJNWbYkPiLb+8kTomcWZzFmioPBORP8Pcg+f0C4j9MxBE7rEYL0bLWGtluI?= =?us-ascii?Q?X7n/aWKn/VMQDnt8wAfAkU3yxdHsH4ukQeoxRmtC/z9Xbch8vgeaSeQ8U4/0?= =?us-ascii?Q?xlfB2J5mE85ow92Kdjr4al6+eATNB10/vzLCO5SrPwMDAtdWe+Bn61j2V580?= =?us-ascii?Q?0h2sD5F4NAFlEXDJaq/nKopYg+0+mkPWkfAbNnauJWBk/0r3J4yR+E0pR15P?= =?us-ascii?Q?SnQoMRn9PHKspkh0AIax6B7HXXUszrKfnDLkRiNn0JVxDvC+p7CP+fnNk7wp?= =?us-ascii?Q?C33SjOJQ9BBOTGyKe+GJR6DGbVN+Ac6kR7YKeAPK0b9KKWfY9fIT6dt4/fqt?= =?us-ascii?Q?0mAELjdQfIVWmDEMdzISdJMpMyqgTkWdPR9fzG1lfWel3RnYJb4cDwMTTRsn?= =?us-ascii?Q?bGO8o02cmD9Ny+sPRqNKHotnfKr8DBIV2AoQS1+mZEIbO5CXo8RCvzVTnJ6W?= =?us-ascii?Q?gCrr+EBxBotAl97e8rvD8ZDzCUs5Fy1l7xw6gUFsu8uOauwC5XPcnvAtDhha?= =?us-ascii?Q?chqaL+sqpfhmLKiHxoauKHvVRMGNsLhfyBsy3A0764v5INVulQXo235+SWAv?= =?us-ascii?Q?SJjjNdwrgsINNugx/5+LjiSvlDVfb9Yqqyt0zsbCN3thhXeLBcQM1j4IoIlN?= =?us-ascii?Q?ooY4HnpwxfxxE3XSmz9HfIpjdRqkqWkYfngS7JQsF1SSXXDVVVhjxKIFT3LH?= =?us-ascii?Q?mLysgLOuspq2N3nE2BTib+n6YGHiSpvk9?= X-Microsoft-Exchange-Diagnostics: 1; YTXPR0101MB2046; 6:Qj49HpDyrG/nVVMSp3ebMASBf8Mnd35Q8B/pxO43zmDKncZU44ltHBmg51GyYzCA8hOqZaDL9XuVFfglRlfvv3Cl08RXs2Q1wxSJArYzl9+llQE15D07C+a0yrQtCYCcpgPD42omeR7bUNL/ExkmJg7gE+ebNMyYWwq4mWZz/MMCivbUsx01sSbPdGTYyoC4syfapu5YaY4ZXQZQpm9/9xR8J+DJjva22xtZDXh3wuTc4PJdBAmUcDFSfZf4BQBEdGJ0GpVn2OdRGx2440UmqIJJam+eDydgCCkwUkfAag0N09wkvD/LS6VoaIq8YOMpdXd96whdnqFnZG0CPyy2UAdnaDWDP0LLZmhhOt7xaUs=; 5:/3MD8Mf9bZ491Mga0k8OEaUv7+yvBPr1KULVv49odyQ96Q2H3mYqQ1ViAQGOAdveT9UWFhxrQL3W+NxPtuG8edUNQYtsgVdDss2GkCXJ1quJWo750fG04ZBM5uAIQwXbvn0hl+181WGGPCvmLo0HrNvRRD06THWC2IIk8sobydQ=; 24:9qgI0RqRDLOxTwjnsqn+eQeH6NtgUMSJhdba7X3uAW8HYX5W5Uoj01mHmIB8KCPjW4Ea9WCJ5I2Y2sSuUvAIF+qTO8CxHTFtwf/coxweuXs=; 7:7gCdV3q25fAhwSxiX+lV4x9/PPAWPUnmCjx628bFhfa3GKeJLxUFw08r0sJemW9VYEw6GptwD3NENSFaYRPsI0ZFdI8nmuxRe9VhHNxmqT425wy5f0TV6FskGbXkV+n5YAX25lcUaG08rauOa2m0qsweLwhURsonJhvEHwxQkw8g1yyopEaB0FFfYrK2SEKUICdSDqmCY8DSlMKJAOaenV5081Bagri8zJBTSos7Z/Az0l1/4h+LjUZEMvqB0GC+ SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2017 02:37:55.5148 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 296b766f-b423-403d-c5fe-08d51ce3bb9a X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR0101MB2046 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Stephen Bates On some servers the BIOS sets up ACS on any valid pci_dev in the system. The kernel has no way of backing this out since the kernel only turns ACS capabilities on. This patch adds a new boot option to the pci kernel parameter called "acs_disable" that will disable ACS. This is useful for PCI peer to peer communication but can cause problems when IOVA isolation is required and an IOMMU is enabled. Use with care. Signed-off-by: Stephen Bates --- Documentation/admin-guide/kernel-parameters.txt | 4 ++++ drivers/pci/pci.c | 23 ++++++++++++++++++++--- drivers/pci/pci.h | 2 +- drivers/pci/probe.c | 4 ++-- 4 files changed, 27 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 0549662..695eb12 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2907,6 +2907,10 @@ earlydump [X86] dump PCI config space before the kernel changes anything off [X86] don't probe for the PCI bus + acs_disable [PCIE] disable access control services. Note + this can interfere with IOVA isolation if an IOMMU + is enabled but can be necessary when doing PCI + peer to peer communication. Use with care. bios [X86-32] force use of PCI BIOS, don't access the hardware directly. Use this if your machine has a non-standard PCI host bridge. diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 6078dfc..ce33608 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -110,6 +110,9 @@ unsigned int pcibios_max_latency = 255; /* If set, the PCIe ARI capability will not be used. */ static bool pcie_ari_disabled; +/* If set, the PCIe ACS capability will be disabled. */ +static bool pci_acs_disable; + /* Disable bridge_d3 for all PCIe ports */ static bool pci_bridge_d3_disable; /* Force bridge_d3 for all PCIe ports */ @@ -1182,7 +1185,7 @@ void pci_restore_state(struct pci_dev *dev) pci_restore_msi_state(dev); /* Restore ACS and IOV configuration state */ - pci_enable_acs(dev); + pci_config_acs(dev); pci_restore_iov_state(dev); dev->state_saved = false; @@ -2821,11 +2824,23 @@ static void pci_std_enable_acs(struct pci_dev *dev) } /** - * pci_enable_acs - enable ACS if hardware support it + * pci_config_acs - configure ACS * @dev: the PCI device */ -void pci_enable_acs(struct pci_dev *dev) +void pci_config_acs(struct pci_dev *dev) { + int pos; + + if (pci_acs_disable) { + pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS); + if (!pos) + return; + + dev_warn_ratelimited(&dev->dev, + "disabling ACS via pci_acs_disable\n"); + pci_write_config_word(dev, pos + PCI_ACS_CTRL, 0); + } + if (!pci_acs_enable) return; @@ -5471,6 +5486,8 @@ static int __init pci_setup(char *str) if (*str && (str = pcibios_setup(str)) && *str) { if (!strcmp(str, "nomsi")) { pci_no_msi(); + } else if (!strcmp(str, "acs_disable")) { + pci_acs_disable = true; } else if (!strcmp(str, "noaer")) { pci_no_aer(); } else if (!strncmp(str, "realloc=", 8)) { diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index a6560c9..16d94d3 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -338,7 +338,7 @@ static inline resource_size_t pci_resource_alignment(struct pci_dev *dev, return resource_alignment(res); } -void pci_enable_acs(struct pci_dev *dev); +void pci_config_acs(struct pci_dev *dev); #ifdef CONFIG_PCIE_PTM void pci_ptm_init(struct pci_dev *dev); diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index ff94b69..86c3299 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2020,8 +2020,8 @@ static void pci_init_capabilities(struct pci_dev *dev) /* Address Translation Services */ pci_ats_init(dev); - /* Enable ACS P2P upstream forwarding */ - pci_enable_acs(dev); + /* Configure ACS P2P upstream forwarding */ + pci_config_acs(dev); /* Precision Time Measurement */ pci_ptm_init(dev);