From patchwork Mon Apr 15 19:33:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 13630590 Received: from EUR03-AM7-obe.outbound.protection.outlook.com (mail-am7eur03on2045.outbound.protection.outlook.com [40.107.105.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 317D1157A7D for ; Mon, 15 Apr 2024 19:34:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.105.45 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713209656; cv=fail; b=ZYMX88ydmcmbjPC0xZvIcckkVW2p5kAnPpnQYJ6DEBvEuOZbgUNanLKB9J7msSvbjyjdqs9zHy6Lgzgzz4/9iBQjmfaUPWtidaFkmhWquF+7EdA2uLQUFgjzNnJvGrec8x23nHbatvfOcGYom1bPcp7xZGJHa8C+xaOaCRO2PqI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713209656; c=relaxed/simple; bh=T5eM7jGli++Q0uN2S+ZvsPFpy84pD1uEfNuEA+XWxBA=; h=From:Date:Subject:Content-Type:Message-Id:References:In-Reply-To: To:Cc:MIME-Version; b=o5rHAFAaQeMUmlbgp3l9n/WG2Pi/PImrHL9ahWkC0VePkDg9W3fFAHo/6XTZRiw4tOdNu6wGeLAvhZXwIybwyyoG4zu7hH3+zsbg0py15XCWWJZIFkrjQz/c7roiWKYP09A0tOar9V032punKwix72xdosDbw9Mo62uzgzzd+nM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nxp.com; spf=pass smtp.mailfrom=nxp.com; dkim=pass (1024-bit key) header.d=nxp.com header.i=@nxp.com header.b=ZjeyPuOM; arc=fail smtp.client-ip=40.107.105.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nxp.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=nxp.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nxp.com header.i=@nxp.com header.b="ZjeyPuOM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WEt4sNSihD5ZDx1VnhXvNru6lXpHLF64bpRR17gsMJAZcmjlkD4NJNsGwXKqj6JJkQPztYoakj/vUyAMYIxfnNRBbVNOR1aexH8bjiGT5EL+sSK5UP4KZYo+LZvqwzcqhVkLuXb/fYfgVJmHVnoyBgs51u/yfMklSfEw9/6n8HuKXFfkYHTyOr0Z2AAo9fGmih8V+Y//ug4Uv12kleNfAK/r06tApKK/4+biaRihS0SQjaTNFyRg0sCFKfMbAZ6GW60Avxu++6D6prDSz+SzW1+Opq+84dH31+Cc4NDbEYkbNgDOTwHYFvuVP4soeZciMcLZFbUOhKvegxB5bF5+Nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=szb01cgagiGs7LgbVEcXCjvASrOLIDPOofiqOm+8tQ4=; b=Bh2dfNR9Rvhi4FvZvMmvQmlecRI1phl/0LwgziO5UmSRx6zCMKksGJRiLobCCwp6ClbvPkoxsrgUqrzMI3+SKE3YJesgc8DsRa4ke0n6ji5s45thl8xoHIIc1N9f/j9CXst29agsOfcrlNKjjnE6w7ZzHf11+wwtE3B51lSS5sRsf/2tj+fhBSVLRlExYTFYptzawOyDiwACQsHaGkb69bb5nxdJ/T/9IAky+2DHFEdlkbr6H9UFTIxVHHcoYUMwQJ8ew3qM4oMiiYUnewwOQi4LJYGTjx0PKtmZ7oW+NvxjyzIOqzlDPxxLqoQM6SGD381LqTZZy5rVSMOrlOKeEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=szb01cgagiGs7LgbVEcXCjvASrOLIDPOofiqOm+8tQ4=; b=ZjeyPuOMURJXfjNb1deisLOZT0Lpcp7uGjgdteCV5M0sYwuC0ICD2k6AtQrCc5yftRCOhNl9rs/XQhAboaQrg82HOzPm7vOPyCkYr2qbxl5vKsECAdZTN2592wGMrUr05ENnSpLdfOA9441eiAniBIXlGc8NRBcNM6t7IC5h1oI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9642.eurprd04.prod.outlook.com (2603:10a6:102:240::14) by PA2PR04MB10279.eurprd04.prod.outlook.com (2603:10a6:102:406::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.43; Mon, 15 Apr 2024 19:34:11 +0000 Received: from PAXPR04MB9642.eurprd04.prod.outlook.com ([fe80::1e67:dfc9:d0c1:fe58]) by PAXPR04MB9642.eurprd04.prod.outlook.com ([fe80::1e67:dfc9:d0c1:fe58%7]) with mapi id 15.20.7452.049; Mon, 15 Apr 2024 19:34:11 +0000 From: Frank Li Date: Mon, 15 Apr 2024 15:33:29 -0400 Subject: [PATCH v6 5/5] PCI: dwc: Add generic MSG TLP support for sending PME_Turn_Off when system suspend Message-Id: <20240415-pme_msg-v6-5-56dad968ad3a@nxp.com> References: <20240415-pme_msg-v6-0-56dad968ad3a@nxp.com> In-Reply-To: <20240415-pme_msg-v6-0-56dad968ad3a@nxp.com> To: Bjorn Helgaas , Jingoo Han , Gustavo Pimentel , Manivannan Sadhasivam , Lorenzo Pieralisi , =?utf-8?q?Krzysztof_Wilczy?= =?utf-8?q?=C5=84ski?= , Rob Herring , Krzysztof Kozlowski , Conor Dooley , imx@lists.linux.dev Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, Frank Li X-Mailer: b4 0.13-dev-e586c X-Developer-Signature: v=1; a=ed25519-sha256; t=1713209628; l=6259; i=Frank.Li@nxp.com; s=20240130; h=from:subject:message-id; bh=T5eM7jGli++Q0uN2S+ZvsPFpy84pD1uEfNuEA+XWxBA=; b=aRstxW3H3QIp5/DrFXUvQCYxpTwwTi3aPojUX/ZFd/ummGxc1M4egr/RNDp4llAAdarDjm0yw onZ7zl5aCWBAK/w2bnv62f1ZLEYNGv39X9aL8KP/5QrA88pSxrk/Lwa X-Developer-Key: i=Frank.Li@nxp.com; a=ed25519; pk=I0L1sDUfPxpAkRvPKy7MdauTuSENRq+DnA+G4qcS94Q= X-ClientProxiedBy: SJ0PR13CA0102.namprd13.prod.outlook.com (2603:10b6:a03:2c5::17) To PAXPR04MB9642.eurprd04.prod.outlook.com (2603:10a6:102:240::14) Precedence: bulk X-Mailing-List: imx@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PAXPR04MB9642:EE_|PA2PR04MB10279:EE_ X-MS-Office365-Filtering-Correlation-Id: a019d729-429f-4868-fd8c-08dc5d8306c9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VnzkHrzpJNivn5AdsutqWLftdPAweTPqTzPuqVy5SPOCn81WW2t0XXjHFUbfF68/TQSlfXRLtdWcChasPC1PvrzBdFctIwJhRrW/+a/Wy7K46j3Q1qGrqGmnNIvH9npaHZ/N/R7bdNcfEvGuFg5XRq39odvuudfH/xU2bEMzBfE6fjhVIn7ySptYp/DyoXd9tmClRPjt0JFXySNgDFDDvjnGXwIl/YDlOvH0TsGeewF5u8zMGrUow1pcU6GF13x3C8aah5qGu5xUfvlvyhkYH1dRFF/eGKgKYGdMINHv1kviJQsihUB3JPzIb88Rz4688/c83Iu9Nux4YXgRp5yJh/b5eudT8JTLYY8QDt9ZRfYuzb7RWLW2TbOv53N7n4UYsGvP/Q+CClF58I+mdUgugQsWYhdBSKHXQDJ9llXnFEHmPb1+Vm2w5nJT+zUTQPWX/N8LjnIPwMi9WNFXQrlpHUjms8vf+gAKznJWfhT25t1CiJfT49MNCWVjOueLQhPGaogepL4lFyrGVZruCY0UNMdK8JUqWXnzWwEAwptJT4Iq8SM+nNGmfrzFGFRPL/HwPH5S+hft0CiZIQo/kpZYfJvV7VKRxk95YlYSC1/Ll6omRJOSyH0p+A33ei2Tlhpr0E9X+xz37mOsgQkQp8/8i/W2DH3GVQwpodCAzW0OskdF3u2Hzk42zzn+EKHlDMOVCyWJp6GfhZOJkU/+ab269plrniZU0NHbYDIEkhUG0I4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9642.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(1800799015)(366007)(376005)(7416005)(52116005)(38350700005)(921011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?DAwen3cAvrq84C4h/ORky+gKN4ZP?= =?utf-8?q?k/e+Q8pJ4KrYfr/sefC8VQ+ZVCgYNsQ/R1np9vxfFPCTo5OlBBTJKus2RjIo8HyTg?= =?utf-8?q?evaSte4qSLx9FiWIhnssg9qnfqLl2fZohUNo842wYT/DB4VbGftHuJg3vEqeQ2KQC?= =?utf-8?q?HutWlb4QgU1BHbIZ/qpObmkj5Lk4Ccu+4NGTyeEGWoEvBERRZf6cbb26f2r42+3R9?= =?utf-8?q?Kqqh+5yKIBbY168Brt6mJJevyrebH4JS7SRYOtDhX5FGFR5wB+XG9pT+sj3LIkocx?= =?utf-8?q?VC78u+VhcnlAx5R/zQIu5IX2/l4Q03SkqqgLI7u8t5NcC/0W6Cg5v66T5EPXScwk2?= =?utf-8?q?geMYIBm2V0hWM0Mhox5vHdy+2SDj9XAZI5jVXTlY3MTfSIjg258WcZeDuIJUxpPYm?= =?utf-8?q?2f+WJf9IWcW+9t/DpUnIcTN3ZaYNDXThqLQkAVBw64h08m2vIHsZOVBRxnQqZoF8o?= =?utf-8?q?uyhrZIuFi2i9ysD5kBl3+CKLR2iJldHC1X2nA0f7qmvetdYb7ZLFbPTJA3BNGU9Bo?= =?utf-8?q?ufRWB7aLtTIUJ+9rDusPkpAdcfAEzB65I3Tp5LNLTQkbBtHEgp6PpiLzXfsOl4LLD?= =?utf-8?q?hFpcs4nOftHyYoBY5vmfd9P3/uwZx6vTQfD7voWjDnMP9OiDyp7JMqtmtHAGH4o+6?= =?utf-8?q?si4xnfQIXMv0bUYF+k+x+4bQnMQAzLp5VhqjbrJ9TI5bOlj0aDDX5WCJmBKaLJy2B?= =?utf-8?q?5hlwuwHVMKlq311wOTwNCwyABsZSnZNIoPHl1uIkV+CxtjrbU0sBSN8tMWXWHOueG?= =?utf-8?q?+qLGKrxTEw5SPc1DQzpAGAO2qiADz873dJZM1/VOEqYTaezNuw8U3bt1ZR+PAYDEh?= =?utf-8?q?ds6Wr9kkE+a3hF7EgupJUfB2dzg87WEJhqt8T58CPVz98IpgUO+HZdbqcJX55Ens6?= =?utf-8?q?FesJSnRceOg7Lwm3omlcESZnABFfKh+89M3rV5IwFziPW+VStDTMUnJoZKi2rhCMT?= =?utf-8?q?eW6IFB1yW/jaWVe/kx6BmBaIp769pi/6kFkbOFq4nNkXs7Tq1VxWzmWlDFV1BZIRg?= =?utf-8?q?Zw82d9PzeNtVttm939Htu2Gd7V580nkj8XmPI6yLmhy4t4kwnqMz8gQoEcVrxRPaN?= =?utf-8?q?+KHTWJaM6i0+EuIHkgAeKmerhR0d5EqyWswBJq3u0jv/CQ0MmnnzLysk+fA4kDLWK?= =?utf-8?q?F5a73oi3x9CDI3H2PMytCcCg1er7kbvId4uvvF9NHeenKB/rjZAVl8wriHF76xsBU?= =?utf-8?q?7WuoWwV0Epa4f0FXR5k51dGN7XchBXufhJeTgVa+DnrMbQFcpeTZEOUTMlb4YOepB?= =?utf-8?q?Fn7VZAJHEYoLoY8F+7QOsSIFicLNHFLVV+uEZinwpWTwXKw53bsM5QCXyjD1ss+z9?= =?utf-8?q?nb9gUr3rnVerkuGitu8MitQcccaFpybJf0djac+EaxlbK5+etL/u3KcTE7zJH9ScJ?= =?utf-8?q?P9zoc/lMZb71Z+CWwgxRDRIGaA/hCs802L7ngpuLQNnPqsipQlhrzQn0H3iueUHjb?= =?utf-8?q?Gj69djS49jXWJFE9PXNl7WFWDCag+WeIcY64+gv1V9eLRMf8UuVWIm9Q=3D?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: a019d729-429f-4868-fd8c-08dc5d8306c9 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9642.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2024 19:34:11.7238 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zgEOQmaRJINeTMfGUR/2iPj+zmuwPe5WSJzQFJxsFEFsdHjaE1WkdYS3whxIjqRaQCAmk/5kAenbEJruxtAOiQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA2PR04MB10279 Instead of relying on the vendor specific implementations to send the PME_Turn_Off message, let's introduce a generic way of sending the message using the MSG TLP. This is achieved by reserving a region for MSG TLP of size 'pci->region_align', at the end of the first IORESOURCE_MEM window of the host bridge. And then sending the PME_Turn_Off message during system suspend with the help of iATU. The reserve space at end is a little bit better than alloc_resource() because the below reasons. - alloc_resource() will allocate space at begin of IORESOURCE_MEM window. There will be a hole when first Endpoint Device (EP) try to alloc a bigger space then 'region_align' size. - Keep EP device's IORESOURCE_MEM space unchange with/without this patch. It should be noted that this generic implementation is optional for the glue drivers and can be overridden by a custom 'pme_turn_off' callback. Signed-off-by: Frank Li --- drivers/pci/controller/dwc/pcie-designware-host.c | 94 ++++++++++++++++++++++- drivers/pci/controller/dwc/pcie-designware.h | 3 + 2 files changed, 93 insertions(+), 4 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index 3a9cb4be22ab2..5001cdf220123 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -398,6 +398,31 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) return 0; } +static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp) +{ + struct dw_pcie *pci = to_dw_pcie_from_pp(pp); + struct resource_entry *win; + struct resource *res; + + win = resource_list_first_type(&pp->bridge->windows, IORESOURCE_MEM); + if (win) { + res = devm_kzalloc(pci->dev, sizeof(*res), GFP_KERNEL); + if (!res) + return; + + /* Reserve last region_align block as message TLP space */ + res->start = win->res->end - pci->region_align + 1; + res->end = win->res->end; + res->name = "msg"; + res->flags = win->res->flags | IORESOURCE_BUSY; + + if (!devm_request_resource(pci->dev, win->res, res)) + pp->msg_res = res; + else + devm_kfree(pci->dev, res); + } +} + int dw_pcie_host_init(struct dw_pcie_rp *pp) { struct dw_pcie *pci = to_dw_pcie_from_pp(pp); @@ -484,6 +509,16 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp) dw_pcie_iatu_detect(pci); + /* + * Allocate the resource for MSG TLP before programming the iATU outbound window in + * dw_pcie_setup_rc(). Since the allocation depends on the value of 'region_align', this has + * to be done after dw_pcie_iatu_detect(). + * + * Glue driver need set use_atu_msg before dw_pcie_host_init() if want MSG TLP feature. + */ + if (pp->use_atu_msg) + dw_pcie_host_request_msg_tlp_res(pp); + ret = dw_pcie_edma_detect(pci); if (ret) goto err_free_msi; @@ -541,6 +576,11 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp) dw_pcie_edma_remove(pci); + if (pp->msg_res) { + release_resource(pp->msg_res); + devm_kfree(pci->dev, pp->msg_res); + } + if (pp->has_msi_ctrl) dw_pcie_free_msi(pp); @@ -702,6 +742,10 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp) atu.pci_addr = entry->res->start - entry->offset; atu.size = resource_size(entry->res); + /* MSG TLB window resource reserve at one of end of IORESOURCE_MEM resource */ + if (pp->msg_res && pp->msg_res->parent == entry->res) + atu.size -= resource_size(pp->msg_res); + ret = dw_pcie_prog_outbound_atu(pci, &atu); if (ret) { dev_err(pci->dev, "Failed to set MEM range %pr\n", @@ -733,6 +777,8 @@ static int dw_pcie_iatu_setup(struct dw_pcie_rp *pp) dev_warn(pci->dev, "Ranges exceed outbound iATU size (%d)\n", pci->num_ob_windows); + pp->msg_atu_index = i; + i = 0; resource_list_for_each_entry(entry, &pp->bridge->dma_ranges) { if (resource_type(entry->res) != IORESOURCE_MEM) @@ -838,11 +884,48 @@ int dw_pcie_setup_rc(struct dw_pcie_rp *pp) } EXPORT_SYMBOL_GPL(dw_pcie_setup_rc); +/* Using message outbound ATU to send out PME_Turn_Off message for dwc PCIe controller */ +static int dw_pcie_pme_turn_off(struct dw_pcie *pci) +{ + struct dw_pcie_ob_atu_cfg atu = { 0 }; + void __iomem *mem; + int ret; + + if (pci->num_ob_windows <= pci->pp.msg_atu_index) + return -EINVAL; + + if (!pci->pp.msg_res) + return -EINVAL; + + atu.code = PCIE_MSG_CODE_PME_TURN_OFF; + atu.routing = PCIE_MSG_TYPE_R_BC; + atu.type = PCIE_ATU_TYPE_MSG; + atu.size = resource_size(pci->pp.msg_res); + atu.index = pci->pp.msg_atu_index; + + atu.cpu_addr = pci->pp.msg_res->start; + + ret = dw_pcie_prog_outbound_atu(pci, &atu); + if (ret) + return ret; + + mem = ioremap(atu.cpu_addr, pci->region_align); + if (!mem) + return -ENOMEM; + + /* A dummy write is converted to a Msg TLP */ + writel(0, mem); + + iounmap(mem); + + return 0; +} + int dw_pcie_suspend_noirq(struct dw_pcie *pci) { u8 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); u32 val; - int ret; + int ret = 0; /* * If L1SS is supported, then do not put the link into L2 as some @@ -854,10 +937,13 @@ int dw_pcie_suspend_noirq(struct dw_pcie *pci) if (dw_pcie_get_ltssm(pci) <= DW_PCIE_LTSSM_DETECT_ACT) return 0; - if (!pci->pp.ops->pme_turn_off) - return 0; + if (pci->pp.ops->pme_turn_off) + pci->pp.ops->pme_turn_off(&pci->pp); + else + ret = dw_pcie_pme_turn_off(pci); - pci->pp.ops->pme_turn_off(&pci->pp); + if (ret) + return ret; ret = read_poll_timeout(dw_pcie_get_ltssm, val, val == DW_PCIE_LTSSM_L2_IDLE, PCIE_PME_TO_L2_TIMEOUT_US/10, diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h index 703b50bc5e0f1..dca5de4c6e877 100644 --- a/drivers/pci/controller/dwc/pcie-designware.h +++ b/drivers/pci/controller/dwc/pcie-designware.h @@ -341,6 +341,9 @@ struct dw_pcie_rp { struct pci_host_bridge *bridge; raw_spinlock_t lock; DECLARE_BITMAP(msi_irq_in_use, MAX_MSI_IRQS); + bool use_atu_msg; + int msg_atu_index; + struct resource *msg_res; }; struct dw_pcie_ep_ops {