From patchwork Wed Mar 27 06:35:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Manivannan Sadhasivam X-Patchwork-Id: 13605619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CAEE0C54E67 for ; Wed, 27 Mar 2024 06:36:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wy1IRMZR3e8xOBtDtj/AwpsAVJYmWJc/ZztDpRw2R2U=; b=Vtfy2+Xmd8gxZm m3CYK7fhOHx8v91j7Juz0F9aVJCpQxae0LP1hmtdiyLwYFbwYYTrIWKKEhO/VM0J9u60FSmFF/CKR /KAYUADufCO4QjuJAXcausakn4QYsHxQXozPpoDntgaDlfReAJIDWKk1sYFZgdTmx60McyrFrwcR9 BMcNKbWM/HqJcI+RkIRGertOI/rB28RgnNtE+cDD2VHzph+rt7kzekfLFkCLltkTCP7pv/Upu3dSr Tn/QbN0hMokWwqQW9+kzqrs0ErkFGW8Xc9OTyf4ZdsZaLwIHLX/BC3WPqRhV7J5l9zEOAzliOOG5A KCalDMdGo55KeAl7XIFg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpMu0-00000007g2j-1DTl; Wed, 27 Mar 2024 06:36:48 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpMtx-00000007g0R-1Byn for linux-rockchip@lists.infradead.org; Wed, 27 Mar 2024 06:36:46 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1e034607879so50652215ad.0 for ; Tue, 26 Mar 2024 23:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1711521405; x=1712126205; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=ms4/cCoO9t4kQQ5ANCu4uj6Ldb1VzG42n8OC1+sLfgU=; b=uXr2qhwKFemG6kGjDSHxOimy2vrsd3Obpbq97eH65rELZwGPPhuqDUl1eOxOd0/aaF dn5tRA+zMuVX0heQIuFMKcIR9bXgD4piVbIlvcROed71KDiiutJKQxZgFHmwkWxSxgKp Hd91bYU70x2U+FKLAkyQN9tRZLBIHpLQ6wkfD5fEtzJ2XULnsJ2fK2z5DeV2J/5245pk IyUlsDmELV5D278xfLTnbyD7Ay2xCrdNkrf2/5ApBWsRSe+ABKLTSiHeFl2jJ401EnQQ HRq74mG7X3UqM/sOfxf9zaeCeBA6uo/bpcNow/Xw6KBPXA/P/izMs47M79BUQ7VQuq13 vGIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711521405; x=1712126205; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ms4/cCoO9t4kQQ5ANCu4uj6Ldb1VzG42n8OC1+sLfgU=; b=cSuIDUsQIXuLyTYrzx4uruKw2n+e/qa2WcJg7oaq6q6GqQazWbpKjvUQww7rYgHByX 1YmiIaQ2QxMwAb83MD3jGHbgD1dzXyvF9feQvv7fUT4C4F4cPz94c0qdNygvEW1O4Y/D j2XWA33sII2OmMnB3osAr2qa2S94WZUxmCXnJ5nhZ0OVHw6guygSRV108CIwaYDUge0d P4RIB8P6Q6anBLETaOa6PZ0gRS1IMTmBGDU9N8s1kHGlj/yBKfULOoXtpGc8zj17Kjwg cnoSQ5wOCpgKh0hWJyVo6g4vO42mCVnBEDWgELy5ubFkiMwiayHWeZ72mZwobNKyGTJ1 Y6nw== X-Forwarded-Encrypted: i=1; AJvYcCWo6Brw6XvT8BG3PYIbzlU5KuiFWUx8V1dar8YknfEMdqp44zXFa0eoX8ypwlBK/8kvGx8DPvaBqBRFnyM3dyqRyJH5xV9IKWU+/AENHgvW/Pgi X-Gm-Message-State: AOJu0YxZKXXlSGyv5msHv0zXYbVybwqR1FUEo8z0xZGiboVnRhdIcQt0 i1KknR3kB9YGUoAo7qK+79HHZ7DUAuzcRMsHvoy79J+2KQU0Rin6LFGHOh917A== X-Google-Smtp-Source: AGHT+IGfb/5qpheudJAFQDoQ+Y66yU71tsgjR+uPsTCQEqq4JR9dsREPelKP5MBglXDbwKKL8InnkA== X-Received: by 2002:a17:902:f707:b0:1e0:2344:e8fa with SMTP id h7-20020a170902f70700b001e02344e8famr507533plo.66.1711521404473; Tue, 26 Mar 2024 23:36:44 -0700 (PDT) Received: from [127.0.1.1] ([117.207.28.168]) by smtp.gmail.com with ESMTPSA id r9-20020a170902be0900b001e14807c7dfsm602553pls.86.2024.03.26.23.36.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 23:36:43 -0700 (PDT) From: Manivannan Sadhasivam Date: Wed, 27 Mar 2024 12:05:47 +0530 Subject: [PATCH v11 1/8] PCI: dwc: ep: Fix DBI access failure for drivers requiring refclk from host MIME-Version: 1.0 Message-Id: <20240327-pci-dbi-rework-v11-1-6f5259f90673@linaro.org> References: <20240327-pci-dbi-rework-v11-0-6f5259f90673@linaro.org> In-Reply-To: <20240327-pci-dbi-rework-v11-0-6f5259f90673@linaro.org> To: Jingoo Han , Gustavo Pimentel , Lorenzo Pieralisi , =?utf-8?q?Krzysztof_Wilczy?= =?utf-8?q?=C5=84ski?= , Rob Herring , Bjorn Helgaas , Marek Vasut , Yoshihiro Shimoda , Thierry Reding , Jonathan Hunter , Kishon Vijay Abraham I , Vidya Sagar , Vignesh Raghavendra , Richard Zhu , Lucas Stach , Shawn Guo , Sascha Hauer , Pengutronix Kernel Team , Fabio Estevam , NXP Linux Team , Minghuan Lian , Mingkai Hu , Roy Zang , Kunihiko Hayashi , Masami Hiramatsu , Kishon Vijay Abraham I , Jesper Nilsson , Srikanth Thokala , Shawn Lin , Heiko Stuebner Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-omap@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Niklas Cassel , linux-arm-kernel@axis.com, linux-rockchip@lists.infradead.org, Manivannan Sadhasivam , Frank Li X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=openpgp-sha256; l=7480; i=manivannan.sadhasivam@linaro.org; h=from:subject:message-id; bh=DDCZoLHduZCUgXgmwVWPg1p7qsh6fab55IAmfI5Ain8=; b=owEBbQGS/pANAwAKAVWfEeb+kc71AcsmYgBmA75VejBfr07ROuibuxbZqYO8QEep5hEFy1PLd iQo+IXhdMqJATMEAAEKAB0WIQRnpUMqgUjL2KRYJ5dVnxHm/pHO9QUCZgO+VQAKCRBVnxHm/pHO 9dHbB/4pmhskP37BwSxqMzzmK2lo7rBUUac8PcmWoW8wwynl21raO/S4FPL7awE92Kf/WrDW9BR RZvUM9Z4LMenjrVKVui+wJ6/tXWnQ+OJAeMwrj/Yz1GsRPwAEAUvkAMg+StfG2iV93wlJTBUanW arD6jN7mVq4BqqBzlVqTWlsen0TVvepRtKLV0AyM/797EF5q1s46EdEJiVrA/B9ld+hUV+yc4+H 7fbrLIoiKbMxJ3DzLmoLHrI+Xkw/bIOdJDr5Sej93a5n2mLLQqKRLhm/Wt+/flW0zD7h1E7MpsG I8lAs1e/d+QJDIVXKO1XqFNTIZfS7zW6wx+IgFuCMfjw6Y4Y X-Developer-Key: i=manivannan.sadhasivam@linaro.org; a=openpgp; fpr=C668AEC3C3188E4C611465E7488550E901166008 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_233645_460559_D0063F7D X-CRM114-Status: GOOD ( 22.96 ) X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+linux-rockchip=archiver.kernel.org@lists.infradead.org The DWC glue drivers requiring an active reference clock from the PCIe host for initializing their PCIe EP core, set a flag called 'core_init_notifier' to let DWC driver know that these drivers need a special attention during initialization. In these drivers, access to the hw registers (like DBI) before receiving the active refclk from host will result in access failure and also could cause a whole system hang. But the current DWC EP driver doesn't honor the requirements of the drivers setting 'core_init_notifier' flag and tries to access the DBI registers during dw_pcie_ep_init(). This causes the system hang for glue drivers such as Tegra194 and Qcom EP as they depend on refclk from host and have set the above mentioned flag. To workaround this issue, users of the affected platforms have to maintain the dependency with the PCIe host by booting the PCIe EP after host boot. But this won't provide a good user experience, since PCIe EP is _one_ of the features of those platforms and it doesn't make sense to delay the whole platform booting due to PCIe requiring active refclk. So to fix this issue, let's move all the DBI access from dw_pcie_ep_init() in the DWC EP driver to the dw_pcie_ep_init_complete() API. This API will only be called by the drivers setting 'core_init_notifier' flag once refclk is received from host. For the rest of the drivers that gets the refclk locally, this API will be called within dw_pcie_ep_init(). Fixes: e966f7390da9 ("PCI: dwc: Refactor core initialization code for EP mode") Co-developed-by: Vidya Sagar Signed-off-by: Vidya Sagar Reviewed-by: Frank Li Reviewed-by: Niklas Cassel Signed-off-by: Manivannan Sadhasivam --- drivers/pci/controller/dwc/pcie-designware-ep.c | 120 ++++++++++++++---------- 1 file changed, 71 insertions(+), 49 deletions(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 5befed2dc02b..c05304eabb89 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -602,11 +602,16 @@ static unsigned int dw_pcie_ep_find_ext_capability(struct dw_pcie *pci, int cap) int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) { struct dw_pcie *pci = to_dw_pcie_from_ep(ep); + struct dw_pcie_ep_func *ep_func; + struct device *dev = pci->dev; + struct pci_epc *epc = ep->epc; unsigned int offset, ptm_cap_base; unsigned int nbars; u8 hdr_type; + u8 func_no; + int i, ret; + void *addr; u32 reg; - int i; hdr_type = dw_pcie_readb_dbi(pci, PCI_HEADER_TYPE) & PCI_HEADER_TYPE_MASK; @@ -617,6 +622,58 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) return -EIO; } + dw_pcie_version_detect(pci); + + dw_pcie_iatu_detect(pci); + + ret = dw_pcie_edma_detect(pci); + if (ret) + return ret; + + if (!ep->ib_window_map) { + ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, + GFP_KERNEL); + if (!ep->ib_window_map) + goto err_remove_edma; + } + + if (!ep->ob_window_map) { + ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, + GFP_KERNEL); + if (!ep->ob_window_map) + goto err_remove_edma; + } + + if (!ep->outbound_addr) { + addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), + GFP_KERNEL); + if (!addr) + goto err_remove_edma; + ep->outbound_addr = addr; + } + + for (func_no = 0; func_no < epc->max_functions; func_no++) { + + ep_func = dw_pcie_ep_get_func_from_ep(ep, func_no); + if (ep_func) + continue; + + ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); + if (!ep_func) + goto err_remove_edma; + + ep_func->func_no = func_no; + ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSI); + ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, + PCI_CAP_ID_MSIX); + + list_add_tail(&ep_func->list, &ep->func_list); + } + + if (ep->ops->init) + ep->ops->init(ep); + offset = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_REBAR); ptm_cap_base = dw_pcie_ep_find_ext_capability(pci, PCI_EXT_CAP_ID_PTM); @@ -651,14 +708,17 @@ int dw_pcie_ep_init_complete(struct dw_pcie_ep *ep) dw_pcie_dbi_ro_wr_dis(pci); return 0; + +err_remove_edma: + dw_pcie_edma_remove(pci); + + return ret; } EXPORT_SYMBOL_GPL(dw_pcie_ep_init_complete); int dw_pcie_ep_init(struct dw_pcie_ep *ep) { int ret; - void *addr; - u8 func_no; struct resource *res; struct pci_epc *epc; struct dw_pcie *pci = to_dw_pcie_from_ep(ep); @@ -666,7 +726,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) struct platform_device *pdev = to_platform_device(dev); struct device_node *np = dev->of_node; const struct pci_epc_features *epc_features; - struct dw_pcie_ep_func *ep_func; INIT_LIST_HEAD(&ep->func_list); @@ -684,26 +743,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ep->ops->pre_init) ep->ops->pre_init(ep); - dw_pcie_version_detect(pci); - - dw_pcie_iatu_detect(pci); - - ep->ib_window_map = devm_bitmap_zalloc(dev, pci->num_ib_windows, - GFP_KERNEL); - if (!ep->ib_window_map) - return -ENOMEM; - - ep->ob_window_map = devm_bitmap_zalloc(dev, pci->num_ob_windows, - GFP_KERNEL); - if (!ep->ob_window_map) - return -ENOMEM; - - addr = devm_kcalloc(dev, pci->num_ob_windows, sizeof(phys_addr_t), - GFP_KERNEL); - if (!addr) - return -ENOMEM; - ep->outbound_addr = addr; - epc = devm_pci_epc_create(dev, &epc_ops); if (IS_ERR(epc)) { dev_err(dev, "Failed to create epc device\n"); @@ -717,23 +756,6 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) if (ret < 0) epc->max_functions = 1; - for (func_no = 0; func_no < epc->max_functions; func_no++) { - ep_func = devm_kzalloc(dev, sizeof(*ep_func), GFP_KERNEL); - if (!ep_func) - return -ENOMEM; - - ep_func->func_no = func_no; - ep_func->msi_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSI); - ep_func->msix_cap = dw_pcie_ep_find_capability(ep, func_no, - PCI_CAP_ID_MSIX); - - list_add_tail(&ep_func->list, &ep->func_list); - } - - if (ep->ops->init) - ep->ops->init(ep); - ret = pci_epc_mem_init(epc, ep->phys_base, ep->addr_size, ep->page_size); if (ret < 0) { @@ -749,25 +771,25 @@ int dw_pcie_ep_init(struct dw_pcie_ep *ep) goto err_exit_epc_mem; } - ret = dw_pcie_edma_detect(pci); - if (ret) - goto err_free_epc_mem; - if (ep->ops->get_features) { epc_features = ep->ops->get_features(ep); if (epc_features->core_init_notifier) return 0; } + /* + * NOTE:- Avoid accessing the hardware (Ex:- DBI space) before this + * step as platforms that implement 'core_init_notifier' feature may + * not have the hardware ready (i.e. core initialized) for access + * (Ex: tegra194). Any hardware access on such platforms result + * in system hang. + */ ret = dw_pcie_ep_init_complete(ep); if (ret) - goto err_remove_edma; + goto err_free_epc_mem; return 0; -err_remove_edma: - dw_pcie_edma_remove(pci); - err_free_epc_mem: pci_epc_mem_free_addr(epc, ep->msi_mem_phys, ep->msi_mem, epc->mem->window.page_size);