From patchwork Fri Aug 12 21:52:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 12942557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ACD39C00140 for ; Fri, 12 Aug 2022 21:54:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=JMVcTtHxFtpAjmZaD7CI+GfcnLRs0r1XrTrhURecls8=; b=yeSWEYxddTjxoE GSlaQPd3GXILGGRKarZ5Pcf8gBK3h3meTuRPrxzUZgGWlraOrATP8lt45rKUti8KKfwy91ZSbjL2k /A4+M/51K4MinCOK+YKKNZEwIBzIU14v0+kHsSIwqDYFHm1E/iKFrRZGpREk9QQAz89XxP2yCcFgO USMf2Lq/FAw7R51CUnSrQKmOpekzdxgTnlq0rfqe0ztnEDZ7ohIatsfAmDk9rrwh37aXA719WBLg+ jycYr4d3imKsMBZyJlrIQaKaWlwP9N8KEPRMtz+5mSfP9scNjGc5GJemoLvX6Ik6/BKLP0Io0CMya c5YmWq6sTm9l9wAWiUog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oMcaf-00DvD4-JS; Fri, 12 Aug 2022 21:53:13 +0000 Received: from mail-am6eur05on2061.outbound.protection.outlook.com ([40.107.22.61] helo=EUR05-AM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oMcaa-00Dv9e-WD for linux-arm-kernel@lists.infradead.org; Fri, 12 Aug 2022 21:53:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LYfHsPyuI7rwa42xUpOMW0BaWJfNV3PKHlnZOIS36LHIUSS8AgyQeN7Jc62leJESNaKrmnvLcKar+wZLwG7wBU8yPgeV1tV01lYEzBFbatqEgenYcLUSiPF409qL/meATcrhVYw8UNybJ6a9FzJsRTfCM3F32sBmsC9rUAGvr5DZiY9WZrSzcvxAWtYSdK/2zW/1T8s9I77nHi9RTKi//GEJtyu9mNen5tQRcLpILi2t/i4idOmuVYtuxyIF7HYPjYDIAK4uCTDc9DerTNanxFDGeBTkNL6fG+q7ayQojKcZbwMQEpFWq4xYnTGWGJ/yFb1uT2Ot1GBfbpoTihn/bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HFJyrv0Xx2Ur5Qdjp6r83x9ircfqsPnRALMElrURjgM=; b=HbtuSNRMBo6axdiw1NPh27Ek1sf6tJMAj58YcWyddLkbEWXgjCInRMZUNY43e/FIUbo4Ef7M761RExswsRIiGY0O63X5MMeXUlgpcqG9dCYNzpvivz5eyhdHnaAJ2jJ3xHcOxWRS+RBA2mZbtFmw7nd5VtpnSN7ezFSf760sGFEnW6LBX12CRM79wIEXTjeZRCRNraD8NnVV0JEc41W7HZejtQg12+JeRBkDIW4QFD7kDWP26IU94Y10nNlV4M+vyeHAPo3cef4U0nZJOQodW/h0AcRpdgiWQb+ug0Fu9o5h8N6dlZcxJbqrVO414DBhkchgntUlA4sjfTvbWPafJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nxp.com; dmarc=pass action=none header.from=nxp.com; dkim=pass header.d=nxp.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HFJyrv0Xx2Ur5Qdjp6r83x9ircfqsPnRALMElrURjgM=; b=Cmpl3sJyYCei67HDzTKtCPlTaFBqkdr4IphW7Oqky2DxxVX47JRFy984QKNOAzMIr3+GRlhCBOMvSRVf7G7bnhQ16lH9bHFSVq2nOAXTFCv62cOhZqsaUxYiXDQ3NDkBo2ttbK9TpKF1q8fj05QDiqIxnzYJ7mSX4YFpJa/6vdY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) by GV1PR04MB9087.eurprd04.prod.outlook.com (2603:10a6:150:22::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5525.10; Fri, 12 Aug 2022 21:53:02 +0000 Received: from PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::c5f1:b708:61db:a004]) by PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::c5f1:b708:61db:a004%6]) with mapi id 15.20.5504.023; Fri, 12 Aug 2022 21:53:01 +0000 From: Frank Li To: maz@kernel.org, tglx@linutronix.de, robh+dt@kernel.org, krzysztof.kozlowski+dt@linaro.org, shawnguo@kernel.org, s.hauer@pengutronix.de, kw@linux.com, bhelgaas@google.com Cc: kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, peng.fan@nxp.com, aisheng.dong@nxp.com, jdmason@kudzu.us, kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, kishon@ti.com, lorenzo.pieralisi@arm.com, ntb@lists.linux.dev, lznuaa@gmail.com Subject: [PATCH v4 0/4] PCI EP driver support MSI doorbell from host Date: Fri, 12 Aug 2022 16:52:38 -0500 Message-Id: <20220812215242.2255824-1-Frank.Li@nxp.com> X-Mailer: git-send-email 2.35.1 X-ClientProxiedBy: BYAPR06CA0064.namprd06.prod.outlook.com (2603:10b6:a03:14b::41) To PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0109858b-fe8d-4b85-d0e7-08da7cad0709 X-MS-TrafficTypeDiagnostic: GV1PR04MB9087:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 14GOLx58O9rU1G5cDS9KUqugOKdAED5AAcKjEk1GDlCW1tjpJ5Sx607/ZjI10nx5eHKtNHXnsktkZuQnTvC5VIldy4tN8LJkWYlWDMyMH13YVv9hkn4S7JKf/2/Klkiudcm4y+8Db62Zt+KSx2Ibi0IMQWmocjDWWfVAk78huf5uA/Jggyxxpv/0SCA5ykQevDDV8KSSQ25+tjixXg3SadNH3icJKFnV4Xi2OQbpY+jfzTQLhDi/eyt2v7V3dZRAmRniek893wbFe/kAH7iYIv25xLpaHBm39Y7Ufjl4DxjaOyoMC32bUCD//vzPCBNiIO6F2vGA+Ol6WL9xi6gvwG9W8LAJcsPumU49xjzTYEA4HoJBnRIrn0xlB7lWZVCFY10FI3xzR70u4xyZLA2m/u/QcuaxOnvCv7sVRdqlz9RoVYov5r+Wot/QRMZfE2UpsXQAvdbL4anzoPDF+t5drQ3QZYs208vzZvxIck9bBvK+oRsDD8OaD7v9E957nYaAcF5IVZuuPPxRy0im3h7/GU82pGe5OWcXvBVxaWAwq/l5pauHTWpjg/wq8bDFZu6NMPe1TzPywy2Z0TI4UR0Cbk8gyvhbo/iJy2LCNFEzOaTJYQnJZ7ETf1WSH6spXZObF7cNa0j7xJg4koDZGm0Ig4dlGgvWyotCG+Ij64yx7lCb7FXcMXZZcs6v6xq9JEKpwRIx/hFNCpJZ08tuC5Vr8J/ysp0cqxXJ41IshOOKs/sOTgC97UI6dCi9rHTlFtcHIEiAc1ymqqvJATfIiF+RPvlvjeioq/4l0FyGreGpHoBl3NsGXfhJTqYvv+9hGNQa3i5u9dUj0aiftdGDQLiEUA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9186.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(396003)(346002)(136003)(39860400002)(376002)(1076003)(2616005)(186003)(83380400001)(38100700002)(6486002)(478600001)(38350700002)(966005)(316002)(66556008)(41300700001)(6666004)(7416002)(5660300002)(52116002)(2906002)(8936002)(26005)(6512007)(66946007)(8676002)(6506007)(4326008)(66476007)(86362001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?tSIj6aXxkVcCs/T7UDBqBDLJMiOQ?= =?utf-8?q?x9dcJRsMOf4+uWjZR8swFVvbmYYYe9UO0JxREjPMOOD15xcFTuducklS1HC4NlFqx?= =?utf-8?q?evVCkZXrbKAZ93RnlxLLOEfRZWFQayWb0JqPxnPyHAlPGUDof8klVh+5coJNnkoU9?= =?utf-8?q?GNDdMlQdJQwMJFPDRlxs6KZWwpzaJpStIu2BP2IEAs6SVINKaRQ365/3hhYwBN850?= =?utf-8?q?WV14M6lgKzY0N6YsE+AsfNV6fZxjuvEbIajSZVodYYgSyjGpmUOSMwbsm6Az9dJOX?= =?utf-8?q?h2/RuVNdXesb6epsIlB/mxHARX2EPnCNmU0jYD+brJEMjXssYxed8vCBPYz6gMZOt?= =?utf-8?q?1X0c1zfwy6Sc9/6ES0vaTjMjn5bO5pLAdYMovREpS/uNxkhoLni6U5BhpSU14kX4H?= =?utf-8?q?3CV7PhxonH3uUMYtlJA7LFeKZbesj//I+pTT8SBtSQEIyiMnYGEefqCOsJUNeLvbA?= =?utf-8?q?t/JG0i6fM33zWCn+09yFqm2FkOp7kHFBRAvEUqqfdkYnK9w3mRFNReN/HvEHxFhzA?= =?utf-8?q?qGnia6VS9McYcPsn0chvUx4B5+UrpV+Hntyk0tqCdH3+eag5u8v5O8QOPJLInZt44?= =?utf-8?q?GudamkU0vqrzMcnHPCE3WLCI5KY63PzBdi+ytnP9Biqt1Fst9GoDskrwXz0pn0PKA?= =?utf-8?q?AAQBDdPDyiW2AaQYf+DX+l5usqbIj12tih9za5hGp/7CzgzmSX2DJEqjxKymw3/YW?= =?utf-8?q?jg9twoAqPFqlfuW1X5iwEkYZKTJM5QWps813NaubJ9yMu3/cuZdVIrn+SkvzUXzKx?= =?utf-8?q?G94kcHc9ttdHUQIOVpUwygs9Cc3Ia2B5TV5quFAqrXczAlU65VfzGWnFQUjEAoikH?= =?utf-8?q?+oq0ZceWHvmvQ/LFhi4hTXGjvizkSkqLBlCB4byxmopzX8L0j6MCiNoFTJEQ5OaNd?= =?utf-8?q?DCkBHnPlmvV4wKx1xQI6kfASI1fOYfEAWePlHtiQ/cKnWnS8nenVZRlxSZLEdnEUq?= =?utf-8?q?byfFIGo0xmbE8TsICfRmF5cEl+DFT4sLofkJ87DEeSmVnCv/ULUNn0wnVYxBfQDoY?= =?utf-8?q?0tCpiE4OL/zCuccdZra2e0uesxSvMiimBevLCYRHaW5IkwhtvStT6Pau08nCvuySk?= =?utf-8?q?SEmyOyoQl19pI4+pUGqZLbl/6Cfgr/9gEKV57/c8GMmZevtT2TW2R+SYbUahhIXto?= =?utf-8?q?Gb7jm+OhkR2MyGlODBwP4Od/ePirAm6smwvY+QgwL4ny3QSBEc0PoelGiGTeF+4zN?= =?utf-8?q?0fCwXy7vTDFNyd8//7W6oexjX6PUHBI7MlI+f2/ES4zlkzfiJX9muck27wVn8WhWo?= =?utf-8?q?JDi/HiSN7Aaa6s8M0Muax9C4zTmXhjIaMTrfmIutRK+Dp7QgHJfI0CgKg1ATQ3Bxm?= =?utf-8?q?GPiN5+Rca+COAtaAiE5/sPbfLtK4jjnu1J6ZmDGGda3erJjJjmZa3QaYJ+c8J902P?= =?utf-8?q?3Sn34ylBgZ9vQUjAmx1d+uxwQUDq0e5zmK102vUq5LMDAd5StafcaXb9usrFd6mHx?= =?utf-8?q?6D/WJSMhbDPppGiVS6Hang+UKNr33dAYc6JFmp8ptlhLRX5lCmgFTXPro3l9cWUTY?= =?utf-8?q?yaeGNTkytX86?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0109858b-fe8d-4b85-d0e7-08da7cad0709 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9186.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Aug 2022 21:53:01.7491 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: z766RuB/Gw/khMdWstSYUMbtJe9bsH8wb9jB21BlEfK0BO47D1YSjY43fEx9PcB2NPFGuDAjpRu5CK1X6DXX7w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR04MB9087 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220812_145309_286450_6C5FEAD4 X-CRM114-Status: GOOD ( 13.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ┌───────┐ ┌──────────┐ │ │ │ │ ┌─────────────┐ │ │ │ PCI Host │ │ MSI │◄┐ │ │ │ │ │ Controller │ │ │ │ │ │ └─────────────┘ └─┼───────┼──────────┼─Bar0 │ │ PCI │ │ Bar1 │ │ Func │ │ Bar2 │ │ │ │ Bar3 │ │ │ │ Bar4 │ │ ├─────────►│ │ └───────┘ └──────────┘ Many PCI controllers provided Endpoint functions. Generally PCI endpoint is hardware, which is not running a rich OS, like linux. But Linux also supports endpoint functions.  PCI Host write BAR space like write to memory. The EP side can't know memory changed by the Host driver. PCI Spec has not defined a standard method to do that.  Only define MSI(x) to let EP notified RC status change. The basic idea is to trigger an IRQ when PCI RC writes to a memory address. That's what MSI controller provided.  EP drivers just need to request a platform MSI interrupt, struct MSI_msg *msg will pass down a memory address and data.  EP driver will map such memory address to one of PCI BAR.  Host just writes such an address to trigger EP side IRQ. If system have gic-its, only need update PCI EP side driver. But i.MX have not chip support gic-its yet. So we have to use MU to simulate a MSI controller. Although only 4 MSI IRQs are simulated, it matched vntb(pci-epf-vntb) network requirement. After enable MSI, ping delay reduce < 1ms from ~8ms IRQchip: imx mu worked as MSI controller: let imx mu worked as MSI controllers. Although IP is not design as MSI controller, we still can use it if limited IRQ number to 4. pcie: endpoint: pci-epf-vntb: add endpoint MSI support Based on ntb-next branch. https://github.com/jonmason/ntb/commits/ntb-next Using MSI as door bell registers This patch is totally independent on previous on. It can be applied to ntb-next seperately. i.MX EP function driver is upstreaming by Richard Zhu. Some dts change missed at this patches. below is reference dts change Fixed irqchip-imx-mu-worked-as-msi-controller according to Marc Zyngier's comments. There are still two important points, which I am not sure. 1. clean irq_set_affinity after platform_msi_create_irq_domain. Some function, like platform_msi_write_msg() is static. so I have to set MSI_FLAG_USE_DEF_CHIP_OPS flags, which will set irq_set_affinity to default one. 2. about comments > + msi_data->msi_domain = platform_msi_create_irq_domain( > + of_node_to_fwnode(msi_data->pdev->dev.of_node), > + &imx_mu_msi_domain_info, > + msi_data->parent); "And you don't get an error due to the fact that you use the same fwnode for both domains without overriding the domain bus token?" I did not understand yet. Fixed static check warning, reported by Dan Carpenter pcie: endpoint: pci-epf-vntb: add endpoint MSI support - Change from v2 to v3 Fixed dt-binding docment check failure Fixed typo a cover letter. Change according Bjorn's comments at patch pcie: endpoint: pci-epf-vntb: add endpoint MSI support - from V1 to V2 Fixed fsl,mu-msi.yaml's problem Fixed irq-imx-mu-msi.c problem according Marc Zyngier's feeback Added a new patch to allow pass down .pm by IRQCHIP_PLATFORM_DRIVER_END --- a/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi @@ -160,5 +160,6 @@ pcieb_ep: pcie_ep@5f010000 { num-ib-windows = <6>; num-ob-windows = <6>; status = "disabled"; + MSI-parent = <&lsio_mu12>; }; --- a/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi @@ -172,6 +172,19 @@ lsio_mu6: mailbox@5d210000 { status = "disabled"; }; + lsio_mu12: mailbox@5d270000 { + compatible = "fsl,imx6sx-mu-MSI"; + msi-controller; + interrupt-controller; + reg = <0x5d270000 0x10000>, /* A side */ + <0x5d300000 0x10000>; /* B side */ + reg-names = "a", "b"; + interrupts = ; + power-domains = <&pd IMX_SC_R_MU_12A>, + <&pd IMX_SC_R_MU_12B>; + power-domain-names = "a", "b"; + }; + Change Log - Change from v3 to v4 Fixed dt-binding document according to Krzysztof Kozlowski's feedback