From patchwork Tue Feb 15 05:38:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 12746513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C99AC433F5 for ; Tue, 15 Feb 2022 05:39:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231736AbiBOFjR (ORCPT ); Tue, 15 Feb 2022 00:39:17 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:40738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233097AbiBOFjQ (ORCPT ); Tue, 15 Feb 2022 00:39:16 -0500 Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-eopbgr130059.outbound.protection.outlook.com [40.107.13.59]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 563A41EC4C for ; Mon, 14 Feb 2022 21:39:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FyQOcD5N6bd61bjiaDykjTHSMTg/ABo/b/32+FdG2HhHHBjwNrQMVUkC9vxRyAbHL1VxUL8l0tN8DH66KQZ2RgdnxLUxJlRZ4mb4PIoX3PoDbjcY02c5Exdrp3usmYenFSqB6y01504ha4WaYLWb1UTboiGjHBBqvdvd2YJ1I8wa7YdkBdrmhtR31PHfLdp7vrWH3owFW8DtHKO86DU8szQKx4ezlw+uDX/DBVAUR6QC5pJfIbpSmFooVLWLUfJbO/dF6SkmcTlZm5yFZ0YtwItlNd39DOkS1asEzqmaImTeaovuPn4J1JkuSDHMGaaICdc+RnA0ICObxG8YXWeOgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O8lJdambt1DZkBjhSD1bu8GRiWtRwsMUN8KTre368As=; b=DdPp4DdugOSDEct7xE6sdbqZbH5VrC+f/33/tiCUPprrrqlueNatLWEPEyXuJR5d7MhSv+kC7V5zPE1hW5jF7kuC0mWRokjfJLBkVbpQShvEcngJO9Y0p49QwJklAOAlO61g80oUD3KqmYRaWUo+6j4TccCa+0CTJ25YkhPWLKjIFTaMBH4fYRVOcNA2v6wO5U+9cQ8n74+vyxPQP5Tez5FLXSRY1Q+jY50lc6eUwoABeZmFBj1l4xsEu2GZbBkaoy6MZLxZN42eyl8BAVwxrksRRGZDOmIjg+fzFy1+OT49vW5P85nk/vq1YmD09mngAzk5fXfg0TBKL09DM52Cfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O8lJdambt1DZkBjhSD1bu8GRiWtRwsMUN8KTre368As=; b=ewyyAuqQZeTvMunZS8yxfTzMIeLz7V05WcocK/tZsEmD9G+pM02BoADo4LMbWGn8bODDXRBxXqoh2SYQ2CVYEYk7SjKwTX9WVdgbN2ElGgX7V93ZzBlY7JeLglP5Tt5U23lvHIkBzm2TiGlrSY5SewdZbLIDSd7C/nybn0KPbmM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) by VI1PR04MB2976.eurprd04.prod.outlook.com (2603:10a6:802:8::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.18; Tue, 15 Feb 2022 05:39:05 +0000 Received: from PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e]) by PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e%6]) with mapi id 15.20.4975.018; Tue, 15 Feb 2022 05:39:05 +0000 From: Frank Li To: linux-pci@vger.kernel.org, kishon@ti.com, lorenzo.pieralisi@arm.com, kw@linux.com, jingoohan1@gmail.com, gustavo.pimentel@synopsys.com, lznuaa@gmail.com, hongxing.zhu@nxp.com Subject: [RFC PATCH 1/4] PCI: designware-ep: Allow pcie_ep_set_bar change inbound map address Date: Mon, 14 Feb 2022 23:38:41 -0600 Message-Id: <20220215053844.7119-2-Frank.Li@nxp.com> X-Mailer: git-send-email 2.24.0.rc1 In-Reply-To: <20220215053844.7119-1-Frank.Li@nxp.com> References: <20220215053844.7119-1-Frank.Li@nxp.com> X-ClientProxiedBy: BYAPR21CA0007.namprd21.prod.outlook.com (2603:10b6:a03:114::17) To PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6cf9c484-a329-40f9-b0b6-08d9f0457af1 X-MS-TrafficTypeDiagnostic: VI1PR04MB2976:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2089; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pDffWdBx845R+jQtXZTFWlYUtvT9tGdq+gaJgyiZbebeeb+KQw3Dy7cqqzzoCz9KOeebU8dTBbDYioFZeusp8FAfe3bQg+gudMk6Yec7iU5PZfYTrkocgY+vnwZ8J8qb/jbdhRLjGBdinPbghWelWwGAd3vypdUUV6XhtkyaQ9XFNPhxEWO/AJO/wRWWTjyZMdJEEjTM+WcHmkZOLxUW62IzCQhk7FhZ3We8y7AzgQC3oDX25lpB68/uHfYXYr0T2k93KnxFzQeWxsur6T9lSxPLOqp3L+ybDn+d2RmJpgmlaZyiN1lCDpIHXZP+tzSlYZ7qnxIy2rh6NaxiDSQLnXytibbzAG/J34XsGazhDm7SEouLb2rp5PxHXOyGQdF+d/7hFLjX+r9uWATg6ksMaaLCqIQHRHLdWGTqQuay5EbtrX2WG7bj9XhYnOL0FtLrfAd/FHUAPZAsLVWims6lF5cL60qvDNWr6nzPIa3KOcTcQzlcPUK8Orz4HAcItH0XKoGpLCQU6tbs5CVSwuHkd2nZ5TIuCqd6CBBsJfhpJY9bBHu7rhO9dcXehnunOoNCoSs1z1HIqsBSKPlD8X7t3quRnjob/AQ8HfCQUEhU3GI3LmlErpNNy79EeRvsnAr6z35ZoaSBfaRXsqQRjPjRk1fsSPxQmw8e1daNSgb34krJLDGV6GBT+FlkCnESw6VfxlqpYiy05GV8OrWOPYX+yw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9186.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(6506007)(5660300002)(26005)(52116002)(38350700002)(186003)(6512007)(83380400001)(38100700002)(86362001)(36756003)(2906002)(8936002)(8676002)(508600001)(66476007)(316002)(2616005)(66556008)(66946007)(6666004)(1076003)(6486002)(6636002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fCreNYRqEJp+GEA2y+gM4vanE5AVhY84q/Dn8fQUbmUVqq1khvbGjeUXShGRU0CuJ+ox5SVtSl7Hpy5CnEsDQgiltl0mAmyXsJdf5x7F3niv0NYSoO4yJVl4rjmG+bNF+/UGnhpU19acC4FBPjMOf7hOH2SKX+E/5iAQ5HzyQpAxMn30UeONaxjGqGUZjPmgKQo3t1/sp9vHgzgZDlj/CciqGit+nU+26EoN/lqBvHMHXzyTEcYFXQky/2+WjjsGTRIF5blpl+Q/yoaIlS6kUfHM00QkYZiyGEpFw2mki4v1ATrAuO9oR1hbrJmDQ+Xd4XWTjwvi/scdcyzGZB0v+DsdcrLiU2Pv8MWYGudjCdwE2wu+XMcJgKygl0Y1/Na+GkiSllea7nfT1JL7RwwkensWXDXQh3ulY66BHBjOmVA6cbHzS84Pr/EjUsYYD5AMQ52XWB+6c0csjmzZReqGroOJPQGcC92C3pksTiZ8V/Hpo1AR0fXvDmRtTTG6j3k27tc3BI55uUHUzoFCuqjn0p+hqWBgEIztvGM17JgwYLSpMpszv+a+KtetKMO86Lo3BVmpsQdn8dcAk2JRwKtaurKsj1YUxpsibgXGvdf/vYTvU9c9mYIf5lwcyr3aL9MX69rSyJvi+fz1rxe/IgR2plTUnaYZdIp9APQ2RL0NXO9U7rxOw0g6SDuY4l+BMMnFSneooB34A8E/rlXN1PJFjR3fYJjMASW2HXJpevSDRFgmXgFfMx6aDPDj6GU3aFUO9i9ZS/cHAmnVJ1J8FAQqHG4hGrB+uQMeqsgNVCN8krdLIYRm8M5VWEgweCvaq9XlKYeuj+h99WCxlxSuSaiESIoeQoPkbUcwOBe341P6Rh/xnUgeJn11xlw00S/xmAfeS5q054O6Tt1vDMF5BzSCkW72OfgoW06EGu7ztT9w8yTAo2/GT076cBsVF1XTwCoyRIhBreofHAPB/bkQ89NoN90IA9uxccFpxgRzEQhjg1HBVdjgQFoLuSPMH8H/UTjcq6+7WMFBXjkS4hXdBZ5rPuvUuabSp4eRNw6TUnvpRhRbsNyrg/KtESU/hT83pG6+iKm0ViylopUFxzLvNnRCGQMG4XLgkUD1IMowJ5wYIxGfiJUaW5TTlU4yPNMv4EwwY5bg85HqmwGnelejOKmV7/BwE8pEqGtdcGc/fKLqd2uC0q7OW02fxVlKAz+NcF/h0TMod2jgXvlkpsxH3TCi4BViYlfcJ9zkDrrxGTo4vRLY1vwPI3MfZin74kUWXpwg2OlC6ENYveo/XFKisPxEYBt3S2jlsLmxwMbmSauEd/kVRIFziYDBot0glWiEJSnQSINPj5kQpZLXOjiYp91DT+vhFxyQ7Dw2OmKZlyZycKC7ZZqn/fbWFNjH75g4FMKj6m+u1TO+aca3vBbPbXgprscFz1v2L5B5SHqOCmGbAaskXcNRiZf0I85AX8r0ATNMpdqz5pcXexx8eEAOa5dpwU4ULaukVfEZa0Aw+SQEE9KwNW0jryYZABOmLTVch8aTPCyFHaEdIZ1d4370YamwaPVwNc2Z5pv6H3yBq5xmkzE67j64m8jFyQwlZzvNkljEVHVAUM9WBJRnmifJoW6PGC+iK5K7LXyxyyQ+Wr6/X5o= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6cf9c484-a329-40f9-b0b6-08d9f0457af1 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9186.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 05:39:05.7736 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UZ9k2C1CR+tUcnxjmXWZOQJnVqHQLDVqBUqSTVq8B20vpbPDFwofGSnZ5MiQnR2P3RT0aVOJpiHgMitCHGUhJg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB2976 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org ntb_transfer will set memory map windows after probe. So the inbound map address need be updated dynamtically. Signed-off-by: Frank Li --- drivers/pci/controller/dwc/pcie-designware-ep.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 998b698f40858..cad5d9ea7cc6c 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -161,7 +161,11 @@ static int dw_pcie_ep_inbound_atu(struct dw_pcie_ep *ep, u8 func_no, u32 free_win; struct dw_pcie *pci = to_dw_pcie_from_ep(ep); - free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows); + if (!ep->bar_to_atu[bar]) + free_win = find_first_zero_bit(ep->ib_window_map, pci->num_ib_windows); + else + free_win = ep->bar_to_atu[bar]; + if (free_win >= pci->num_ib_windows) { dev_err(pci->dev, "No free inbound window\n"); return -EINVAL; @@ -215,6 +219,7 @@ static void dw_pcie_ep_clear_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no, dw_pcie_disable_atu(pci, atu_index, DW_PCIE_REGION_INBOUND); clear_bit(atu_index, ep->ib_window_map); ep->epf_bar[bar] = NULL; + ep->bar_to_atu[bar] = 0; } static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no, @@ -244,6 +249,9 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no, if (ret) return ret; + if (ep->epf_bar[bar]) + return 0; + dw_pcie_dbi_ro_wr_en(pci); dw_pcie_writel_dbi2(pci, reg, lower_32_bits(size - 1)); From patchwork Tue Feb 15 05:38:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 12746514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1B08C433F5 for ; Tue, 15 Feb 2022 05:39:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234136AbiBOFjV (ORCPT ); Tue, 15 Feb 2022 00:39:21 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:40786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234123AbiBOFjU (ORCPT ); Tue, 15 Feb 2022 00:39:20 -0500 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10061.outbound.protection.outlook.com [40.107.1.61]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFD261EC6D for ; Mon, 14 Feb 2022 21:39:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K4ODvL7OHwsD/hsRjbcF5QjFepaqKTA4RZbnnDsUQ5DcL9aMquAh1aGDWikcIYRZAbzyuUNkSLc/PYHmYrPU/XkPVwfENaSpPkNQLn1WaVeP7l14BFnZFGNrdhxNK5uqXBT08xRNL5AVXzatF22gEuRZ0zNTD4HXX1zANzGtiuZDNXTQd8zk7XpAjCTwfVlSLAmkIsWGH23LVGT9JvhWVXjPqINQxZB0g2XQJ+r3VmXdm+fbny5chd1SwWIw/d7D35fLGCPRWuBYnfktXLBCktMt7dV9DKe3QGr8VUuX+Elx49qU9RcM4HrmThRInv6hylAfhiROZWGT98aFYg+Kkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V8s3psJlSNwnPzoLzfkQFszGNXas8HAemFD8x+GvoHU=; b=hXfxZxWiOYNFJ3q9eUFXlCQR7VdIONMnfAgd/Ipxp3LeN1CGaVpKJPgWa7ff3HsAC1tqdMlNEKQpDrU/UZo670djVc4oXUQkw/Tvmfpyi73L9HRQRzcgsyW6Vv7n99EOGIyI+wemwfbUkN1UkhuhGaLTLHdVbJDMFfAqhZPiGOWeZ907IqqnfJMzI/sbVkn7nqyI9QOpT/rQcWAncCc5sls8b6nPuOVqza7igSQqNomMe6th4f72IibcKuo0T8yGZrpq2cuB2E9jydpXFS6itxnkyoRTaPbqnRvVDmfvai4nNyUJFOIVeHUWdmy4Ma5fl/pBdNTr7NiLu/YqZe1ZnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V8s3psJlSNwnPzoLzfkQFszGNXas8HAemFD8x+GvoHU=; b=CtVxHwuhKRh/jaANPlTiaJWxtIV9pDW7iMLpNQr2i8R4KLx7YMFb9s+TDIuxr5FJQ9RUKWoZNRJUvt+KOlcJGKPO60oZXWeLLQo1XZWHnPfsikjevUqMNan8chgc/yEeQazRrqsRja05NSqEAi1resaiwoIL0WKjtC0UxHiFl+o= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) by VI1PR04MB2976.eurprd04.prod.outlook.com (2603:10a6:802:8::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.18; Tue, 15 Feb 2022 05:39:08 +0000 Received: from PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e]) by PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e%6]) with mapi id 15.20.4975.018; Tue, 15 Feb 2022 05:39:08 +0000 From: Frank Li To: linux-pci@vger.kernel.org, kishon@ti.com, lorenzo.pieralisi@arm.com, kw@linux.com, jingoohan1@gmail.com, gustavo.pimentel@synopsys.com, lznuaa@gmail.com, hongxing.zhu@nxp.com Subject: [RFC PATCH 2/4] NTB: epf: Added more flexible memory map method Date: Mon, 14 Feb 2022 23:38:42 -0600 Message-Id: <20220215053844.7119-3-Frank.Li@nxp.com> X-Mailer: git-send-email 2.24.0.rc1 In-Reply-To: <20220215053844.7119-1-Frank.Li@nxp.com> References: <20220215053844.7119-1-Frank.Li@nxp.com> X-ClientProxiedBy: BYAPR21CA0007.namprd21.prod.outlook.com (2603:10b6:a03:114::17) To PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9489153c-9a58-4871-85a7-08d9f0457c6a X-MS-TrafficTypeDiagnostic: VI1PR04MB2976:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:381; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 40vZlxYCYFX2N/0f77F8pdAqiqwlWLTLlxcTDOlwgT1aeN4pBDeY/cRyCKS3vO4zsyKgpnwgNFZBO/ja9P068vGooFcDo+KxI4TNZLAh1LJj2Vz/kmvJLRauyc2ryKhaiAdjTn1AsSXGc8+j2oWG8JhSFaHXFO8cPXKDW4ypUFkrYUaN+h0tAUF0exF7vDyACBO0jJSZRMwuVCMJsofVeeg/9VGf3eseTA5jR3odimzrH+SszD0Q++LY5HWpcotwM2PXUPtXnZ1d1pQ9tPvhIfIa2d0mLSItcEdfyjoQyIqjdeG+NPweWyvG/0TqNQIbxZYxzm04jAyLOEDogcIFr3xCsnZtuY0j33x47SUB67FVGL0z3MUeEew5U8gzTCxSpxQtUNqEpSHYLTT2n8HLSk7pajyRR/7yMHT1tSocpaa/rHY/qqyV+ELuTVlManSJYo4Ef3Y2VlVkB61RTSOyzfiUoUKDXH8sU2SAnf3sMhBbSeCpeIV1vZ//bVC0Mh0WarO6sWDnXumdgwOS+y+zo+IugOKk8xPqTKimG83tGN1STAuTT/VBfrASNaPa7NgbnNK2JTmmY+6rVXQrOQb4wgzFUwPHXiQpfgBXiBrdTolcK/36lH4q0r7jf/A6hHg1DHx4FP8TqvPWdS6x1c/w/L9X9qYXnHwlfS13UmISNMx0WuVimp4tB/BHY+hn0nfSU0NS3KXrkhQNHE4oFZvVhg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9186.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(6506007)(5660300002)(26005)(52116002)(38350700002)(186003)(6512007)(83380400001)(38100700002)(86362001)(36756003)(2906002)(8936002)(8676002)(508600001)(66476007)(316002)(2616005)(66556008)(66946007)(6666004)(1076003)(6486002)(6636002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RHK05TZ2/CUbESqoEcy5yMoJI7N/AOGycYF5xWWOGE1RR1vdCHEYWGvwZA0mxGLHlCkN5hTbx535WKbVJmTddpSY47yLwGJ3gisiDtsfrtVcA90RWiEchNvgr1vJrIxTVDracRzI9exKw6zVzT4WNCtagROZKHJ/wa4sAdZGgNoFxUN+6NmY39rnqozWuMnmn9VZb5X4BgY/n5NRXSgrrKmSghmQuu7wtOfP6/uUts2Yco4nzTRdNjIARDfM0zsw7rffpePtNnBLdvuORgAgerh5SQJPwBqSf4brbXA/v1paPmF+8zos5JNAQBRRxasDRUkFXzovlw/x4cncTcCUe1Ogo4q5HTPEokQsMXfb5jfJ52GVuAptKot9WdIHxb+LJl8UqBwp20It27TyGIV2YnKcn0dkAQ7kjIrGwGFecFqfrx56s+BZFXoLmJ2O5jQ1q9Zz6aw05MIeKJDC8R9QLUfURjBGLYBURT9uP731ixMrB156SmHGo4zd525/ufQ4qavjqE+cajiDUaDaePeSy4QlYogUNBv9Z/f6272mhE+PnycpHp0XzGWQafloU7bv5MeM+AxO5eRAyYcvSWK+iV7b7Ge3m9l6JdSaoIwzOmzianFUvxdqU3BmFUmlRXv3Me3naTIQSP2kMPMVDVTOgiey/E3TJVukK8ZZH/1y+37DMTVAo1dOWj3d7+mL0yF5QvKlfHSjmgdkVsMRaz+u58WDn4RLF1FvVscnO9E3CCp2ov9L8yXYBCMlixS4sXz+HwtKqufBS43VZPw0T84VbUyJcmn03YcUeDZLe/lNuS9IbtdNm2NXfMv1N0RLiYphjvdTSRD57LUyJogQ8uYlgCWxgXxNtPSD/e2RNOSxoyxZ54jgnbpz1SZ5pTs5ihcaOpKBBcC7EfGU+wL/kcuFZBLJSyJYl/FhPSWgPgL0JOIi+p7qoWwKznNiWYaR28X3v4vB5cWrrATOhIs0YjCs4iRQ3O2SBDMdsgJ50a3wkcj7ufnl3DHKuLLel2LSyLoUMWEeREIPAZJ6XtrSb7Csq8mVNlJad9gKTZFfFszsq4m1AhFlrf9guwdKDZ9bH2akerIdipBETktF4iMXCZV2/PjmhzJkECSqp981pi49+rYk+9xTLIoU+qRQd9pBMHlTwW5XhkL3H2BSq08u0Zkb0ofOc9zp1DgyTGesCcZvQDFSpTii1FgnH0V8Ze6swGKrWkcL/1RM+OuI143+vKiJYK0r3hfnrEyCnAt5oZT7bE/2JCW3Mbj1DFJtKZlZrZJLux0JgMM7uw9AWiThrOZFafCNEXt8zGvNr9pnx+BV8sxy8KWTGm1ibbWIqlWzAwkEMnfjPQ5RikOjTtJiKy/pUC0kMua6U8KwaX3sCYNPi53AGL9NvjAfMcvnr3i+nWf7SwpDzlgrg4JP68yh8QeBA5giiH62Cb/WeeScf+72U86//D7Kjzs1yuYYKONcmCwvF4Cl2/IGdtaJ0jnjna9yag+tZo0x2Zx5PCbVb181n3WI+dJUlJPLS69zJ+L3iHz4E0rF49145GTl/nCVlsMKqV08dBdAkhSFk8p1yb/j99Y+M4m3T8eChDwyq8HyZMcLQs1/RyF2EszDBgrLunWJe97+hqyYmDimfb9qvppImis= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9489153c-9a58-4871-85a7-08d9f0457c6a X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9186.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 05:39:08.1963 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yn9tIsbqWj0w8tXqizrm1VAKJgkcT6Q3aIOXtScq7R4vM542lqfrXILd2Ijew01rJK2JyL0tWxT5VVpwxLMLkA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB2976 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Supported below memory map method bar 0: config and spad data bar 2: door bell bar 4: memory map windows Signed-off-by: Frank Li --- drivers/ntb/hw/epf/ntb_hw_epf.c | 48 ++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 13 deletions(-) diff --git a/drivers/ntb/hw/epf/ntb_hw_epf.c b/drivers/ntb/hw/epf/ntb_hw_epf.c index b019755e4e21b..3ece49cb18ffa 100644 --- a/drivers/ntb/hw/epf/ntb_hw_epf.c +++ b/drivers/ntb/hw/epf/ntb_hw_epf.c @@ -45,7 +45,6 @@ #define NTB_EPF_MIN_DB_COUNT 3 #define NTB_EPF_MAX_DB_COUNT 31 -#define NTB_EPF_MW_OFFSET 2 #define NTB_EPF_COMMAND_TIMEOUT 1000 /* 1 Sec */ @@ -67,6 +66,7 @@ struct ntb_epf_dev { enum pci_barno ctrl_reg_bar; enum pci_barno peer_spad_reg_bar; enum pci_barno db_reg_bar; + enum pci_barno mw_bar; unsigned int mw_count; unsigned int spad_count; @@ -92,6 +92,8 @@ struct ntb_epf_data { enum pci_barno peer_spad_reg_bar; /* BAR that contains Doorbell region and Memory window '1' */ enum pci_barno db_reg_bar; + /* BAR that contains memory windows*/ + enum pci_barno mw_bar; }; static int ntb_epf_send_command(struct ntb_epf_dev *ndev, u32 command, @@ -411,7 +413,7 @@ static int ntb_epf_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx, return -EINVAL; } - bar = idx + NTB_EPF_MW_OFFSET; + bar = idx + ndev->mw_bar; mw_size = pci_resource_len(ntb->pdev, bar); @@ -453,7 +455,7 @@ static int ntb_epf_peer_mw_get_addr(struct ntb_dev *ntb, int idx, if (idx == 0) offset = readl(ndev->ctrl_reg + NTB_EPF_MW1_OFFSET); - bar = idx + NTB_EPF_MW_OFFSET; + bar = idx + ndev->mw_bar; if (base) *base = pci_resource_start(ndev->ntb.pdev, bar) + offset; @@ -565,6 +567,7 @@ static int ntb_epf_init_pci(struct ntb_epf_dev *ndev, struct pci_dev *pdev) { struct device *dev = ndev->dev; + size_t spad_sz, spad_off; int ret; pci_set_drvdata(pdev, ndev); @@ -599,10 +602,16 @@ static int ntb_epf_init_pci(struct ntb_epf_dev *ndev, goto err_dma_mask; } - ndev->peer_spad_reg = pci_iomap(pdev, ndev->peer_spad_reg_bar, 0); - if (!ndev->peer_spad_reg) { - ret = -EIO; - goto err_dma_mask; + if (ndev->peer_spad_reg_bar) { + ndev->peer_spad_reg = pci_iomap(pdev, ndev->peer_spad_reg_bar, 0); + if (!ndev->peer_spad_reg) { + ret = -EIO; + goto err_dma_mask; + } + } else { + spad_sz = 4 * readl(ndev->ctrl_reg + NTB_EPF_SPAD_COUNT); + spad_off = readl(ndev->ctrl_reg + NTB_EPF_SPAD_OFFSET); + ndev->peer_spad_reg = ndev->ctrl_reg + spad_off + spad_sz; } ndev->db_reg = pci_iomap(pdev, ndev->db_reg_bar, 0); @@ -657,6 +666,7 @@ static int ntb_epf_pci_probe(struct pci_dev *pdev, enum pci_barno peer_spad_reg_bar = BAR_1; enum pci_barno ctrl_reg_bar = BAR_0; enum pci_barno db_reg_bar = BAR_2; + enum pci_barno mw_bar = BAR_2; struct device *dev = &pdev->dev; struct ntb_epf_data *data; struct ntb_epf_dev *ndev; @@ -671,17 +681,16 @@ static int ntb_epf_pci_probe(struct pci_dev *pdev, data = (struct ntb_epf_data *)id->driver_data; if (data) { - if (data->peer_spad_reg_bar) - peer_spad_reg_bar = data->peer_spad_reg_bar; - if (data->ctrl_reg_bar) - ctrl_reg_bar = data->ctrl_reg_bar; - if (data->db_reg_bar) - db_reg_bar = data->db_reg_bar; + peer_spad_reg_bar = data->peer_spad_reg_bar; + ctrl_reg_bar = data->ctrl_reg_bar; + db_reg_bar = data->db_reg_bar; + mw_bar = data->mw_bar; } ndev->peer_spad_reg_bar = peer_spad_reg_bar; ndev->ctrl_reg_bar = ctrl_reg_bar; ndev->db_reg_bar = db_reg_bar; + ndev->mw_bar = mw_bar; ndev->dev = dev; ntb_epf_init_struct(ndev, pdev); @@ -729,6 +738,14 @@ static const struct ntb_epf_data j721e_data = { .ctrl_reg_bar = BAR_0, .peer_spad_reg_bar = BAR_1, .db_reg_bar = BAR_2, + .mw_bar = BAR_2, +}; + +static const struct ntb_epf_data mx8_data = { + .ctrl_reg_bar = BAR_0, + .peer_spad_reg_bar = BAR_0, + .db_reg_bar = BAR_2, + .mw_bar = BAR_4, }; static const struct pci_device_id ntb_epf_pci_tbl[] = { @@ -737,6 +754,11 @@ static const struct pci_device_id ntb_epf_pci_tbl[] = { .class = PCI_CLASS_MEMORY_RAM << 8, .class_mask = 0xffff00, .driver_data = (kernel_ulong_t)&j721e_data, }, + { + PCI_DEVICE(PCI_VENDOR_ID_FREESCALE, 0x0809), + .class = PCI_CLASS_MEMORY_RAM << 8, .class_mask = 0xffff00, + .driver_data = (kernel_ulong_t)&mx8_data, + }, { }, }; From patchwork Tue Feb 15 05:38:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 12746515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4961C433F5 for ; Tue, 15 Feb 2022 05:39:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234111AbiBOFj0 (ORCPT ); Tue, 15 Feb 2022 00:39:26 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:40878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233097AbiBOFjZ (ORCPT ); Tue, 15 Feb 2022 00:39:25 -0500 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10085.outbound.protection.outlook.com [40.107.1.85]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D78E7201BB for ; Mon, 14 Feb 2022 21:39:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dqc/40ZubxcUCZjaW8XTiU5pDa/OI2TIRjQ8sEdUkXyxyuZ/WnqGhZb5tpmA6fUNARajXLjke7Yc4ex1bU2DIIkvss6v1denRIrHJA3XUZrLmaC8pJ29bbUYx5SGjV5oxYH9mOt6YPd3FnCnMPtFN9r4blC3idabURYcqETVih+Wr3A+ptMPdOoR+9NRpmaSl2+O7S+6JhshuPHeezeFjV4t9H3XchtwikEFJXRV0DhZi2O/NcDJKn36echquC9PFbBiAz3wvOXK3W0pezfz82YG11xjDGuyQW2KsoSctF4y4tpX8iHRivye73HsbVEK/vhKPXDO0/JYtn1970Mlqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1wMaFPhFTzVv2szbrY/tq80kfRy3S+WTmskEOA8k0ls=; b=aoApvANPWzsvTyMK4uQ5AofN6U3rO+EQOjRTZ2RfxVmvDdT2pGp97Np+/+lyf7Fzegqkz30zquUVeACxldGBWjqpBShh41GYd6f+KMGZzCjrKght1x9kcARKWFGD763psRMiV908jJN0flGSqQsUD/ezF5dagsumvK5hdgiz3Wc4I9rs2wuyx8BAvffjaEDOoxbfis2AXMLZN4Opn2xImndHptnXyKjyPSfPd2lHhXJtf3/ElQ+8MeiqN9VTzdfOzEAe8iXXhvHx/AD4STqhNxCnkkb7gF4EGKCuYHG97PhW7bSWLrOy99uPgZZhawIMXIhRRoBfWWSIPsSmkN6MaQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1wMaFPhFTzVv2szbrY/tq80kfRy3S+WTmskEOA8k0ls=; b=tENLn7FQwNoDETAF8TOgEwzriCu1v6/D+31S1k0FSnvpKtEOHtXsc8MRZCnhBLkXRM+UbozII5v5d003jCNuTzZniPLoR7XxQRFQvooyRaX4ZvtQ3sFM1lsKcdc659uy1yOL6vpdqc7epAo8CgnM1Ez4I+YHXluWf4QlbZ2dPu0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) by VI1PR04MB2976.eurprd04.prod.outlook.com (2603:10a6:802:8::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.18; Tue, 15 Feb 2022 05:39:10 +0000 Received: from PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e]) by PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e%6]) with mapi id 15.20.4975.018; Tue, 15 Feb 2022 05:39:10 +0000 From: Frank Li To: linux-pci@vger.kernel.org, kishon@ti.com, lorenzo.pieralisi@arm.com, kw@linux.com, jingoohan1@gmail.com, gustavo.pimentel@synopsys.com, lznuaa@gmail.com, hongxing.zhu@nxp.com Subject: [RFC PATCH 3/4] NTB: EPF: support NTB transfer between PCI RC and EP connection Date: Mon, 14 Feb 2022 23:38:43 -0600 Message-Id: <20220215053844.7119-4-Frank.Li@nxp.com> X-Mailer: git-send-email 2.24.0.rc1 In-Reply-To: <20220215053844.7119-1-Frank.Li@nxp.com> References: <20220215053844.7119-1-Frank.Li@nxp.com> X-ClientProxiedBy: BYAPR21CA0007.namprd21.prod.outlook.com (2603:10b6:a03:114::17) To PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 778449d6-ac98-432b-ba59-08d9f0457dd2 X-MS-TrafficTypeDiagnostic: VI1PR04MB2976:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:23; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2nh45aYy0tAkEcMQwxqbkJlFOf62KScvpmDPB19G0evNvDOch5jxS7F3u1rXmkNLzJcwyNqvCm1punMxAfADYT7GZ28BcV2wWWpmgcycbmYwfgm0N39F/0sqQmZlESg0Fj9dclw81lQiLYRXoSzOGin2uwfFAqYrYRtYW2R11z1KuP3kUmxh7T1rUFZ3DeeDJnwFl6zTn1SqxIrUgg6MbS1s9P7qOUt+kHsnbIHj4NmBYqt8kC4AW+Q+TMcp66rq8s47GURGyx9ZMWXBEiMy8IAW98PcRS1l+IkjMUENbGmqpooEbwki6uzdzlFth6kDZYgSoN0FRvfbBEEZuEcgs7oWwQKi/v3jQev48ANKcTMlEyx+O52fMt9SOfsXuiQ+Nfl+nLhcai18JZ+eBjy/OVaeep1JC/Nu7W4pSwQ7y893awbOBcNcvvi13a5xVTFDN9qinNOy7jggrYAKyt+EesYP7N07uFayYAF1e8Tezrx5jHRco2v6RSG0VfPS2iVULNa061h03GJT4UBJurZ1fOTjSH1zfhYEhW5mfVEMpD4/hE6sMntxVJbaD9onltvtmeTidRCTgZPtkzI/NNqKQwEs+A5vuvJQxLPUgBoPQmvzLe0s/jw5dNs6ylpKTqS+xkSddoQs7LfvfKKIELpnK24H/2ze9Ip+hyPR+fhJu2aVNyg9nVLw9oNNh0CpFLtPxb+BFldlVVirQJtKeDhHtQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9186.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(6506007)(5660300002)(26005)(52116002)(38350700002)(186003)(6512007)(83380400001)(38100700002)(86362001)(36756003)(30864003)(2906002)(8936002)(8676002)(508600001)(66476007)(316002)(2616005)(66556008)(66946007)(6666004)(1076003)(6486002)(6636002)(579004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?kN4pQWQ1DqigVBbdhTAp2XrQow7Z?= =?utf-8?q?tXj/skuYIL+8Uv6zB1rBJg/ky87Ojm9xC7tCeyX7a/1WkAwy9nDSZKAphoWQ9tFh2?= =?utf-8?q?im1bJRumtAIRTM4ba7C5g6S6ScpOYShl0NBV/i6FgTHFiGlbalIYOmAfDW/cbghv2?= =?utf-8?q?TJ3Nli/1s8fCTW55+a3TANJDkMwL71XNZQ7u7HRbT41PoVj9x3tKLa3zKdu0ZW9jF?= =?utf-8?q?htfjg/KhIkpMf7Fh0Ox2haTysFiOXZONC14qqDmRcWOcZPWP/PpTtJJXtFfQO5Zii?= =?utf-8?q?WnokkPfLqGARRWpzpfKrSzfNTsH4GkLnxv79cxUp/qsyKLRGsg+a3vhFGsueGgQkD?= =?utf-8?q?iK5BZeQCyGJtIYLNGAPxkKT0JbMX/GOp0ySuZA751XoEMPoAf17lcxn2SQn/joZEl?= =?utf-8?q?NF2PaTFXSZu0pe9vVqSpO2RHNdSo1Ss16x7ZdDahhNPLX6SAvnP0qPi5UhUPDfAph?= =?utf-8?q?JKHn7N3peM1ZkuuxZk/1QN9BkBvZFLHt7MXTtlipVBtaa+rJlXQr3GpVZUDT9izWI?= =?utf-8?q?tsW9NQ7rMoOJouVQaR7gSgOpoQ2YnQUg0J2CgbUU8rcy1VixotuNiwchPw82FqTJL?= =?utf-8?q?REiYpqsgXOyyIOc2TNbmLEAIVG+OMMBGrU5HxRp+T+9Tf261PvRdSDhqkmIyzZrOx?= =?utf-8?q?m8ZX5OCkZnS5BCFIsLP81ZOmc0TlAKMhjgA6jdTkTXqD3PIdYlJpAp52xKpStty42?= =?utf-8?q?BaVyo+Z5Gw7YmZjmKluFKE0PFsYA3tIxJN6o+82L4U49ctH83eKDdK2EgNfW0JE+W?= =?utf-8?q?rOGEE+mgRB4iDHl+XJ1k7MpqxVhbTeUW2xXzlrlPS856e/cd+EVcJIKXNdOhoy3el?= =?utf-8?q?dFLpWmUQ0h5nvZKNIuyhYfewtavmjjt6jm0w639XJRKihFnrn4nZE/dWfla96fUV0?= =?utf-8?q?nDAAeTW6BnFeTq3CMxvV7fhn66wu1I7oZnXmApaIs4GN6kPtQlsfq/YlHOAadrGKE?= =?utf-8?q?mr945b92IStelzlhq4MuLQPlP2Cg1hERrchqMKai5Mr8NivjWDvYdCA6kiP5CLH6D?= =?utf-8?q?N+ZR8UZZuuTxq2T9A+c7BsTRghrO7HAiyb++Oi4VaeB7QQkr2RnYbeBVswcIADpbP?= =?utf-8?q?FjnhBN1OEAcWEufRodGJEh6bkseAMCYLEtjRoYhHtApVN4RzS09heXeWkM3IIiUNW?= =?utf-8?q?uXyzeOlh77mRiovdEsGU1D6sENLruXWOew5m4egzvNJKfvHCsdtdIODTHZw4pT42I?= =?utf-8?q?xYr0No2muAzCr4zRNv+ulb+RfFrIJT+hLPaZqI3u2qJkd3XBEWrmK7FwZndUqLhhB?= =?utf-8?q?90v/oF/V3HXLfkq874d9/LnYvfsRh+BPexiO3L+xOevqO5lT05jZ80RcG3UxFlPE1?= =?utf-8?q?pW6eLHyl8+tS0s617oWwLY0R86ZbOpw4fUShrJ4Nb8m+YiATR/5cIl9IaGf20zKe8?= =?utf-8?q?JLsQBKXYjsdse7ReVy+QvXzCK5aZoIS7+hqd8L6rKOLaqvpZe0MqGTxcRInBurLOL?= =?utf-8?q?AR9fmVj4rsAZj4jZ2F/7jHy/43SvWxtXOOUJ/sZmlxaZefSkbYWJVpPHYY6xA8z5r?= =?utf-8?q?9+a88vNObjPDxEs0TyPncaJvWz1VCHOrtsz6rDlpQsTqRP0hxwF+IZI=3D?= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 778449d6-ac98-432b-ba59-08d9f0457dd2 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9186.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 05:39:10.7612 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6BwvLmGoZtBKSVp9JJIVMdsMkkngrP+VWa7K+FYY9CQS0vKjtkr55ozWlqdDCAuCOb4b5tmhUkj4Lh3YY9MDqQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB2976 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add NTB function driver and virtual PCI BUS and Virtual NTB driver to implement communication between PCIe RC and PCIe EP devices ┌────────────┐ ┌─────────────────────────────────────┐ │ │ │ │ ├────────────┤ │ ┌──────────────┤ │ NTB │ │ │ NTB │ │ NetDev │ │ │ NetDev │ ├────────────┤ │ ├──────────────┤ │ NTB │ │ │ NTB │ │ Transfer │ │ │ Transfer │ ├────────────┤ │ ├──────────────┤ │ │ │ │ │ │ PCI NTB │ │ │ │ │ EPF │ │ │ │ │ Driver │ │ │ PCI Virtual │ │ │ ├───────────────┐ │ NTB Driver │ │ │ │ PCI EP NTB │◄────►│ │ │ │ │ FN Driver │ │ │ ├────────────┤ ├───────────────┤ ├──────────────┤ │ │ │ │ │ │ │ PCI BUS │ ◄─────► │ PCI EP BUS │ │ Virtual PCI │ │ │ PCI │ │ │ BUS │ └────────────┘ └───────────────┴──────┴──────────────┘ PCI RC PCI EP This driver include 3 part: 1 PCI EP NTB function driver 2 Virtual PCI bus 3 PCI virutal NTB driver, which is loaded only by above virtual pci bus Signed-off-by: Frank Li --- drivers/pci/endpoint/functions/Kconfig | 11 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-vntb.c | 1425 +++++++++++++++++ 3 files changed, 1437 insertions(+) create mode 100644 drivers/pci/endpoint/functions/pci-epf-vntb.c diff --git a/drivers/pci/endpoint/functions/Kconfig b/drivers/pci/endpoint/functions/Kconfig index 5f1242ca2f4e4..362555b024e8f 100644 --- a/drivers/pci/endpoint/functions/Kconfig +++ b/drivers/pci/endpoint/functions/Kconfig @@ -25,3 +25,14 @@ config PCI_EPF_NTB device tree. If in doubt, say "N" to disable Endpoint NTB driver. + +config PCI_EPF_VNTB + tristate "PCI Endpoint NTB driver" + depends on PCI_ENDPOINT + select CONFIGFS_FS + help + Select this configuration option to enable the Non-Transparent + Bridge (NTB) driver for PCI Endpoint. NTB driver implements NTB + between PCI host and PCIe Endpoint. + + If in doubt, say "N" to disable Endpoint NTB driver. diff --git a/drivers/pci/endpoint/functions/Makefile b/drivers/pci/endpoint/functions/Makefile index 96ab932a537a2..5c13001deaba1 100644 --- a/drivers/pci/endpoint/functions/Makefile +++ b/drivers/pci/endpoint/functions/Makefile @@ -5,3 +5,4 @@ obj-$(CONFIG_PCI_EPF_TEST) += pci-epf-test.o obj-$(CONFIG_PCI_EPF_NTB) += pci-epf-ntb.o +obj-$(CONFIG_PCI_EPF_VNTB) += pci-epf-vntb.o diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c new file mode 100644 index 0000000000000..ebf7e243eefa4 --- /dev/null +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c @@ -0,0 +1,1425 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Endpoint Function Driver to implement Non-Transparent Bridge functionality + * Between PCI RC and EP + * + * Copyright (C) 2020 Texas Instruments + * Copyright (C) 2022 NXP + * + * Based on pci-epf-ntb.c + * Author: Frank Li + * Author: Kishon Vijay Abraham I + */ + +/** + * +------------+ +---------------------------------------+ + * | | | | + * +------------+ | +--------------+ + * | NTB | | | NTB | + * | NetDev | | | NetDev | + * +------------+ | +--------------+ + * | NTB | | | NTB | + * | Transfer | | | Transfer | + * +------------+ | +--------------+ + * | | | | | + * | PCI NTB | | | | + * | EPF | | | | + * | Driver | | | PCI Virtual | + * | | +---------------+ | NTB Driver | + * | | | PCI EP NTB |<------>| | + * | | | FN Driver | | | + * +------------+ +---------------+ +--------------+ + * | | | | | | + * | PCI BUS | <-----> | PCI EP BUS | | Virtual PCI | + * | | PCI | | | BUS | + * +------------+ +---------------+--------+--------------+ + * PCI RC PCI EP + */ + +#include +#include +#include +#include + +#include +#include +#include + +static struct workqueue_struct *kpcintb_workqueue; + +#define COMMAND_CONFIGURE_DOORBELL 1 +#define COMMAND_TEARDOWN_DOORBELL 2 +#define COMMAND_CONFIGURE_MW 3 +#define COMMAND_TEARDOWN_MW 4 +#define COMMAND_LINK_UP 5 +#define COMMAND_LINK_DOWN 6 + +#define COMMAND_STATUS_OK 1 +#define COMMAND_STATUS_ERROR 2 + +#define LINK_STATUS_UP BIT(0) + +#define SPAD_COUNT 64 +#define DB_COUNT 4 +#define NTB_MW_OFFSET 2 +#define DB_COUNT_MASK GENMASK(15, 0) +#define MSIX_ENABLE BIT(16) +#define MAX_DB_COUNT 32 +#define MAX_MW 4 + +#define VNTB_VID 0x1957 +#define VNTB_PID 0x080A + +enum epf_ntb_bar { + BAR_CONFIG, + BAR_DB, + BAR_MW0, + BAR_MW1, + BAR_MW2, +}; + +/* + * +--------------------------------------------------+ Base + * | | + * | | + * | | + * | Common Control Register | + * | | + * | | + * | | + * +-----------------------+--------------------------+ Base+span_offset + * | | | + * | Peer Span Space | Span Space | + * | | | + * | | | + * +-----------------------+--------------------------+ Base+span_offset + * | | | +span_count * 4 + * | | | + * | Span Space | Peer Span Space | + * | | | + * +-----------------------+--------------------------+ + * Virtual PCI Pcie Endpoint + * NTB Driver NTB Driver + */ +struct epf_ntb_ctrl { + u32 command; + u32 argument; + u16 command_status; + u16 link_status; + u32 topology; + u64 addr; + u64 size; + u32 num_mws; + u32 reserved; + u32 spad_offset; + u32 spad_count; + u32 db_entry_size; + u32 db_data[MAX_DB_COUNT]; + u32 db_offset[MAX_DB_COUNT]; +} __packed; + +struct epf_ntb { + struct ntb_dev ntb; + struct pci_epf *epf; + struct config_group group; + + u32 num_mws; + u32 db_count; + u32 spad_count; + u64 mws_size[MAX_MW]; + u64 db; + u32 vbus_number; + + bool linkup; + u32 spad_size; + + enum pci_barno epf_ntb_bar[6]; + + struct epf_ntb_ctrl *reg; + + phys_addr_t epf_db_phy; + void __iomem *epf_db; + + phys_addr_t vpci_mw_phy[MAX_MW]; + void __iomem *vpci_mw_addr[MAX_MW]; + + struct delayed_work cmd_handler; +}; + +#define to_epf_ntb(epf_group) container_of((epf_group), struct epf_ntb, group) +#define ntb_ndev(__ntb) container_of(__ntb, struct epf_ntb, ntb) + +static struct pci_epf_header epf_ntb_header = { + .vendorid = PCI_ANY_ID, + .deviceid = PCI_ANY_ID, + .baseclass_code = PCI_BASE_CLASS_MEMORY, + .interrupt_pin = PCI_INTERRUPT_INTA, +}; + +/** + * epf_ntb_link_up() - Raise link_up interrupt to Virtual Host + * @ntb: NTB device that facilitates communication between HOST and VHOST + * @link_up: true or false indicating Link is UP or Down + * + * Once NTB function in HOST invoke ntb_link_enable(), + * this NTB function driver will trigger a link event to vhost. + */ +static int epf_ntb_link_up(struct epf_ntb *ntb, bool link_up) +{ + if (link_up) + ntb->reg->link_status |= LINK_STATUS_UP; + else + ntb->reg->link_status &= ~LINK_STATUS_UP; + + ntb_link_event(&ntb->ntb); + return 0; +} + +/** + * epf_ntb_configure_mw() - Configure the Outbound Address Space for vhost + * to access the memory window of host + * @ntb: NTB device that facilitates communication between host and vhost + * @mw: Index of the memory window (either 0, 1, 2 or 3) + * + * EP Outbound Window + * +--------+ +-----------+ + * | | | | + * | | | | + * | | | | + * | | | | + * | | +-----------+ + * | Virtual| | Memory Win| + * | NTB | -----------> | | + * | Driver | | | + * | | +-----------+ + * | | | | + * | | | | + * +--------+ +-----------+ + * VHost PCI EP + */ +static int epf_ntb_configure_mw(struct epf_ntb *ntb, u32 mw) +{ + phys_addr_t phys_addr; + u8 func_no, vfunc_no; + u64 addr, size; + int ret = 0; + + phys_addr = ntb->vpci_mw_phy[mw]; + addr = ntb->reg->addr; + size = ntb->reg->size; + + func_no = ntb->epf->func_no; + vfunc_no = ntb->epf->vfunc_no; + + ret = pci_epc_map_addr(ntb->epf->epc, func_no, vfunc_no, phys_addr, addr, size); + if (ret) + dev_err(&ntb->epf->epc->dev, + "intf: Failed to map memory window %d address\n", mw); + return ret; +} + +/** + * epf_ntb_teardown_mw() - Teardown the configured OB ATU + * @ntb: NTB device that facilitates communication between HOST and vHOST + * @mw: Index of the memory window (either 0, 1, 2 or 3) + * + * Teardown the configured OB ATU configured in epf_ntb_configure_mw() using + * pci_epc_unmap_addr() + */ +static void epf_ntb_teardown_mw(struct epf_ntb *ntb, u32 mw) +{ + pci_epc_unmap_addr(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no, + ntb->vpci_mw_phy[mw]); +} + +/** + * epf_ntb_cmd_handler() - Handle commands provided by the NTB Host + * @work: work_struct for the epf_ntb_epc + * + * Workqueue function that gets invoked for the two epf_ntb_epc + * periodically (once every 5ms) to see if it has received any commands + * from NTB host. The host can send commands to configure doorbell or + * configure memory window or to update link status. + */ +static void epf_ntb_cmd_handler(struct work_struct *work) +{ + struct epf_ntb_ctrl *ctrl; + u32 command, argument; + struct epf_ntb *ntb; + struct device *dev; + int ret; + int i; + + ntb = container_of(work, struct epf_ntb, cmd_handler.work); + + for (i = 1; i < ntb->db_count; i++) { + if (readl(ntb->epf_db + i * 4)) { + if (readl(ntb->epf_db + i * 4)) + ntb->db |= 1 << (i - 1); + + ntb_db_event(&ntb->ntb, i); + writel(0, ntb->epf_db + i * 4); + } + } + + ctrl = ntb->reg; + command = ctrl->command; + if (!command) + goto reset_handler; + argument = ctrl->argument; + + ctrl->command = 0; + ctrl->argument = 0; + + ctrl = ntb->reg; + dev = &ntb->epf->dev; + + switch (command) { + case COMMAND_CONFIGURE_DOORBELL: + ctrl->command_status = COMMAND_STATUS_OK; + break; + case COMMAND_TEARDOWN_DOORBELL: + ctrl->command_status = COMMAND_STATUS_OK; + break; + case COMMAND_CONFIGURE_MW: + ret = epf_ntb_configure_mw(ntb, argument); + if (ret < 0) + ctrl->command_status = COMMAND_STATUS_ERROR; + else + ctrl->command_status = COMMAND_STATUS_OK; + break; + case COMMAND_TEARDOWN_MW: + epf_ntb_teardown_mw(ntb, argument); + ctrl->command_status = COMMAND_STATUS_OK; + break; + case COMMAND_LINK_UP: + ntb->linkup = true; + ret = epf_ntb_link_up(ntb, true); + if (ret < 0) + ctrl->command_status = COMMAND_STATUS_ERROR; + else + ctrl->command_status = COMMAND_STATUS_OK; + goto reset_handler; + case COMMAND_LINK_DOWN: + ntb->linkup = false; + ret = epf_ntb_link_up(ntb, false); + if (ret < 0) + ctrl->command_status = COMMAND_STATUS_ERROR; + else + ctrl->command_status = COMMAND_STATUS_OK; + break; + default: + dev_err(dev, "intf UNKNOWN command: %d\n", command); + break; + } + +reset_handler: + queue_delayed_work(kpcintb_workqueue, &ntb->cmd_handler, + msecs_to_jiffies(5)); +} + +/** + * epf_ntb_config_sspad_bar_clear() - Clear Config + Self scratchpad BAR + * @ntb_epc: EPC associated with one of the HOST which holds peer's outbound + * address. + * + * Clear BAR0 of EP CONTROLLER 1 which contains the HOST1's config and + * self scratchpad region (removes inbound ATU configuration). While BAR0 is + * the default self scratchpad BAR, an NTB could have other BARs for self + * scratchpad (because of reserved BARs). This function can get the exact BAR + * used for self scratchpad from epf_ntb_bar[BAR_CONFIG]. + * + * Please note the self scratchpad region and config region is combined to + * a single region and mapped using the same BAR. Also note HOST2's peer + * scratchpad is HOST1's self scratchpad. + */ +static void epf_ntb_config_sspad_bar_clear(struct epf_ntb *ntb) +{ + struct pci_epf_bar *epf_bar; + enum pci_barno barno; + + barno = ntb->epf_ntb_bar[BAR_CONFIG]; + epf_bar = &ntb->epf->bar[barno]; + + pci_epc_clear_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar); +} + +/** + * epf_ntb_config_sspad_bar_set() - Set Config + Self scratchpad BAR + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + * Map BAR0 of EP CONTROLLER 1 which contains the HOST1's config and + * self scratchpad region. + * + * Please note the self scratchpad region and config region is combined to + * a single region and mapped using the same BAR. + */ +static int epf_ntb_config_sspad_bar_set(struct epf_ntb *ntb) +{ + struct pci_epf_bar *epf_bar; + enum pci_barno barno; + u8 func_no, vfunc_no; + struct device *dev; + int ret; + + dev = &ntb->epf->dev; + func_no = ntb->epf->func_no; + vfunc_no = ntb->epf->vfunc_no; + barno = ntb->epf_ntb_bar[BAR_CONFIG]; + epf_bar = &ntb->epf->bar[barno]; + + ret = pci_epc_set_bar(ntb->epf->epc, func_no, vfunc_no, epf_bar); + if (ret) { + dev_err(dev, "inft: Config/Status/SPAD BAR set failed\n"); + return ret; + } + return 0; +} + +/** + * epf_ntb_config_spad_bar_free() - Free the physical memory associated with + * config + scratchpad region + * @ntb: NTB device that facilitates communication between HOST and vHOST + */ +static void epf_ntb_config_spad_bar_free(struct epf_ntb *ntb) +{ + enum pci_barno barno; + + barno = ntb->epf_ntb_bar[BAR_CONFIG]; + pci_epf_free_space(ntb->epf, ntb->reg, barno, 0); +} + +/** + * epf_ntb_config_spad_bar_alloc() - Allocate memory for config + scratchpad + * region + * @ntb: NTB device that facilitates communication between HOST1 and HOST2 + * + * Allocate the Local Memory mentioned in the above diagram. The size of + * CONFIG REGION is sizeof(struct epf_ntb_ctrl) and size of SCRATCHPAD REGION + * is obtained from "spad-count" configfs entry. + */ +static int epf_ntb_config_spad_bar_alloc(struct epf_ntb *ntb) +{ + size_t align; + enum pci_barno barno; + struct epf_ntb_ctrl *ctrl; + u32 spad_size, ctrl_size; + u64 size; + struct pci_epf *epf = ntb->epf; + struct device *dev = &epf->dev; + u32 spad_count; + void *base; + int i; + const struct pci_epc_features *epc_features = pci_epc_get_features(epf->epc, + epf->func_no, + epf->vfunc_no); + barno = ntb->epf_ntb_bar[BAR_CONFIG]; + size = epc_features->bar_fixed_size[barno]; + align = epc_features->align; + + if ((!IS_ALIGNED(size, align))) + return -EINVAL; + + spad_count = ntb->spad_count; + + ctrl_size = sizeof(struct epf_ntb_ctrl); + spad_size = 2 * spad_count * 4; + + if (!align) { + ctrl_size = roundup_pow_of_two(ctrl_size); + spad_size = roundup_pow_of_two(spad_size); + } else { + ctrl_size = ALIGN(ctrl_size, align); + spad_size = ALIGN(spad_size, align); + } + + if (!size) + size = ctrl_size + spad_size; + else if (size < ctrl_size + spad_size) + return -EINVAL; + + base = pci_epf_alloc_space(epf, size, barno, align, 0); + if (!base) { + dev_err(dev, "intf: Config/Status/SPAD alloc region fail\n"); + return -ENOMEM; + } + + ntb->reg = base; + + ctrl = ntb->reg; + ctrl->spad_offset = ctrl_size; + + ctrl->spad_count = spad_count; + ctrl->num_mws = ntb->num_mws; + ntb->spad_size = spad_size; + + ctrl->db_entry_size = 4; + + for (i = 0; i < ntb->db_count; i++) { + ntb->reg->db_data[i] = 1 + i; + ntb->reg->db_offset[i] = 0; + } + + return 0; +} + +/** + * epf_ntb_configure_interrupt() - Configure MSI/MSI-X capaiblity + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + * Configure MSI/MSI-X capability for each interface with number of + * interrupts equal to "db_count" configfs entry. + */ +static int epf_ntb_configure_interrupt(struct epf_ntb *ntb) +{ + const struct pci_epc_features *epc_features; + bool msix_capable, msi_capable; + u8 func_no, vfunc_no; + struct device *dev; + u32 db_count; + int ret; + + dev = &ntb->epf->dev; + + epc_features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no); + msix_capable = epc_features->msix_capable; + msi_capable = epc_features->msi_capable; + + if (!(msix_capable || msi_capable)) { + dev_err(dev, "MSI or MSI-X is required for doorbell\n"); + return -EINVAL; + } + + func_no = ntb->epf->func_no; + vfunc_no = ntb->epf->vfunc_no; + + db_count = ntb->db_count; + if (db_count > MAX_DB_COUNT) { + dev_err(dev, "DB count cannot be more than %d\n", MAX_DB_COUNT); + return -EINVAL; + } + + ntb->db_count = db_count; + + if (msi_capable) { + ret = pci_epc_set_msi(ntb->epf->epc, func_no, vfunc_no, 16); + if (ret) { + dev_err(dev, "intf: MSI configuration failed\n"); + return ret; + } + } + + return 0; +} + +/** + * epf_ntb_db_bar_init() - Configure Doorbell window BARs + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + */ +static int epf_ntb_db_bar_init(struct epf_ntb *ntb) +{ + const struct pci_epc_features *epc_features; + u32 align; + struct device *dev = &ntb->epf->dev; + int ret; + struct pci_epf_bar *epf_bar; + void __iomem *mw_addr; + enum pci_barno barno; + size_t size = 4 * ntb->db_count; + + epc_features = pci_epc_get_features(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no); + align = epc_features->align; + + if (size < 128) + size = 128; + + if (align) + size = ALIGN(size, align); + else + size = roundup_pow_of_two(size); + + barno = ntb->epf_ntb_bar[BAR_DB]; + + mw_addr = pci_epf_alloc_space(ntb->epf, size, barno, align, 0); + if (!mw_addr) { + dev_err(dev, "intf: Failed to allocate OB address\n"); + return -ENOMEM; + } + + ntb->epf_db = mw_addr; + + epf_bar = &ntb->epf->bar[barno]; + + ret = pci_epc_set_bar(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no, epf_bar); + if (ret) { + dev_err(dev, "intf: DoorBell BAR set failed\n"); + goto err_alloc_peer_mem; + } + return ret; + +err_alloc_peer_mem: + pci_epc_mem_free_addr(ntb->epf->epc, epf_bar->phys_addr, mw_addr, epf_bar->size); + return -1; +} + +/** + * epf_ntb_db_bar_clear() - Clear doorbell BAR and free memory + * allocated in peers outbound address space + * @ntb: NTB device that facilitates communication between HOST and vHOST + */ +static void epf_ntb_db_bar_clear(struct epf_ntb *ntb) +{ + enum pci_barno barno; + + barno = ntb->epf_ntb_bar[BAR_DB]; + pci_epf_free_space(ntb->epf, ntb->epf_db, barno, 0); + pci_epc_clear_bar(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no, + &ntb->epf->bar[barno]); +} + +/** + * epf_ntb_mw_bar_init() - Configure Memory window BARs + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + */ +static int epf_ntb_mw_bar_init(struct epf_ntb *ntb) +{ + int ret = 0; + int i; + u64 size; + enum pci_barno barno; + struct device *dev = &ntb->epf->dev; + + for (i = 0; i < ntb->num_mws; i++) { + + size = ntb->mws_size[i]; + + barno = ntb->epf_ntb_bar[BAR_MW0 + i]; + + ntb->epf->bar[barno].barno = barno; + ntb->epf->bar[barno].size = size; + ntb->epf->bar[barno].addr = 0; + ntb->epf->bar[barno].phys_addr = 0; + ntb->epf->bar[barno].flags |= upper_32_bits(size) ? + PCI_BASE_ADDRESS_MEM_TYPE_64 : + PCI_BASE_ADDRESS_MEM_TYPE_32; + + ret = pci_epc_set_bar(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no, + &ntb->epf->bar[barno]); + if (ret) { + dev_err(dev, "intf: MW set failed\n"); + goto err_alloc_mem; + } + + /* allocate epc outbound memory windows to vpci vntb device */ + ntb->vpci_mw_addr[i] = pci_epc_mem_alloc_addr(ntb->epf->epc, + &ntb->vpci_mw_phy[i], + size); + if (!ntb->vpci_mw_addr[i]) { + dev_err(dev, "Failed to allocate source address\n"); + goto err_alloc_mem; + } + } + + return ret; +err_alloc_mem: + return ret; +} + +/** + * epf_ntb_mw_bar_clear() - Clear Memory window BARs + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + */ +static void epf_ntb_mw_bar_clear(struct epf_ntb *ntb) +{ + enum pci_barno barno; + int i; + + for (i = 0; i < ntb->num_mws; i++) { + barno = ntb->epf_ntb_bar[BAR_MW0 + i]; + pci_epc_clear_bar(ntb->epf->epc, + ntb->epf->func_no, + ntb->epf->vfunc_no, + &ntb->epf->bar[barno]); + + pci_epc_mem_free_addr(ntb->epf->epc, + ntb->vpci_mw_phy[i], + ntb->vpci_mw_addr[i], + ntb->mws_size[i]); + } +} + +/** + * epf_ntb_epc_destroy() - Cleanup NTB EPC interface + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + * Wrapper for epf_ntb_epc_destroy_interface() to cleanup all the NTB interfaces + */ +static void epf_ntb_epc_destroy(struct epf_ntb *ntb) +{ + pci_epc_remove_epf(ntb->epf->epc, ntb->epf, 0); + pci_epc_put(ntb->epf->epc); +} + +/** + * epf_ntb_init_epc_bar() - Identify BARs to be used for each of the NTB + * constructs (scratchpad region, doorbell, memorywindow) + * @ntb: NTB device that facilitates communication between HOST and vHOST + * + */ +static int epf_ntb_init_epc_bar(struct epf_ntb *ntb) +{ + const struct pci_epc_features *epc_features; + enum pci_barno barno; + enum epf_ntb_bar bar; + struct device *dev; + u32 num_mws; + int i; + + barno = BAR_0; + num_mws = ntb->num_mws; + dev = &ntb->epf->dev; + epc_features = pci_epc_get_features(ntb->epf->epc, ntb->epf->func_no, ntb->epf->vfunc_no); + + /* These are required BARs which are mandatory for NTB functionality */ + for (bar = BAR_CONFIG; bar <= BAR_MW0; bar++, barno++) { + barno = pci_epc_get_next_free_bar(epc_features, barno); + if (barno < 0) { + dev_err(dev, "intf: Fail to get NTB function BAR\n"); + return barno; + } + ntb->epf_ntb_bar[bar] = barno; + } + + /* These are optional BARs which don't impact NTB functionality */ + for (bar = BAR_MW1, i = 1; i < num_mws; bar++, barno++, i++) { + barno = pci_epc_get_next_free_bar(epc_features, barno); + if (barno < 0) { + ntb->num_mws = i; + dev_dbg(dev, "BAR not available for > MW%d\n", i + 1); + } + ntb->epf_ntb_bar[bar] = barno; + } + + return 0; +} + +/** + * epf_ntb_epc_init() - Initialize NTB interface + * @ntb: NTB device that facilitates communication between HOST and vHOST2 + * + * Wrapper to initialize a particular EPC interface and start the workqueue + * to check for commands from host. This function will write to the + * EP controller HW for configuring it. + */ +static int epf_ntb_epc_init(struct epf_ntb *ntb) +{ + u8 func_no, vfunc_no; + struct pci_epc *epc; + struct pci_epf *epf; + struct device *dev; + int ret; + + epf = ntb->epf; + dev = &epf->dev; + epc = epf->epc; + func_no = ntb->epf->func_no; + vfunc_no = ntb->epf->vfunc_no; + + ret = epf_ntb_config_sspad_bar_set(ntb); + if (ret) { + dev_err(dev, "intf: Config/self SPAD BAR init failed"); + return ret; + } + + ret = epf_ntb_configure_interrupt(ntb); + if (ret) { + dev_err(dev, "intf: Interrupt configuration failed\n"); + goto err_config_interrupt; + } + + ret = epf_ntb_db_bar_init(ntb); + if (ret) { + dev_err(dev, "intf: DB BAR init failed\n"); + goto err_db_bar_init; + } + + ret = epf_ntb_mw_bar_init(ntb); + if (ret) { + dev_err(dev, "intf: MW BAR init failed\n"); + goto err_mw_bar_init; + } + + if (vfunc_no <= 1) { + ret = pci_epc_write_header(epc, func_no, vfunc_no, epf->header); + if (ret) { + dev_err(dev, "intf: Configuration header write failed\n"); + goto err_write_header; + } + } + + INIT_DELAYED_WORK(&ntb->cmd_handler, epf_ntb_cmd_handler); + queue_work(kpcintb_workqueue, &ntb->cmd_handler.work); + + return 0; + +err_write_header: + epf_ntb_mw_bar_clear(ntb); +err_mw_bar_init: + epf_ntb_db_bar_clear(ntb); +err_db_bar_init: +err_config_interrupt: + epf_ntb_config_sspad_bar_clear(ntb); + + return ret; +} + + +/** + * epf_ntb_epc_cleanup() - Cleanup all NTB interfaces + * @ntb: NTB device that facilitates communication between HOST1 and HOST2 + * + * Wrapper to cleanup all NTB interfaces. + */ +static void epf_ntb_epc_cleanup(struct epf_ntb *ntb) +{ + epf_ntb_db_bar_clear(ntb); + epf_ntb_mw_bar_clear(ntb); +} + +#define EPF_NTB_R(_name) \ +static ssize_t epf_ntb_##_name##_show(struct config_item *item, \ + char *page) \ +{ \ + struct config_group *group = to_config_group(item); \ + struct epf_ntb *ntb = to_epf_ntb(group); \ + \ + return sprintf(page, "%d\n", ntb->_name); \ +} + +#define EPF_NTB_W(_name) \ +static ssize_t epf_ntb_##_name##_store(struct config_item *item, \ + const char *page, size_t len) \ +{ \ + struct config_group *group = to_config_group(item); \ + struct epf_ntb *ntb = to_epf_ntb(group); \ + u32 val; \ + int ret; \ + \ + ret = kstrtou32(page, 0, &val); \ + if (ret) \ + return ret; \ + \ + ntb->_name = val; \ + \ + return len; \ +} + +#define EPF_NTB_MW_R(_name) \ +static ssize_t epf_ntb_##_name##_show(struct config_item *item, \ + char *page) \ +{ \ + struct config_group *group = to_config_group(item); \ + struct epf_ntb *ntb = to_epf_ntb(group); \ + int win_no; \ + \ + sscanf(#_name, "mw%d", &win_no); \ + \ + return sprintf(page, "%lld\n", ntb->mws_size[win_no - 1]); \ +} + +#define EPF_NTB_MW_W(_name) \ +static ssize_t epf_ntb_##_name##_store(struct config_item *item, \ + const char *page, size_t len) \ +{ \ + struct config_group *group = to_config_group(item); \ + struct epf_ntb *ntb = to_epf_ntb(group); \ + struct device *dev = &ntb->epf->dev; \ + int win_no; \ + u64 val; \ + int ret; \ + \ + ret = kstrtou64(page, 0, &val); \ + if (ret) \ + return ret; \ + \ + if (sscanf(#_name, "mw%d", &win_no) != 1) \ + return -EINVAL; \ + \ + if (ntb->num_mws < win_no) { \ + dev_err(dev, "Invalid num_nws: %d value\n", ntb->num_mws); \ + return -EINVAL; \ + } \ + \ + ntb->mws_size[win_no - 1] = val; \ + \ + return len; \ +} + +static ssize_t epf_ntb_num_mws_store(struct config_item *item, + const char *page, size_t len) +{ + struct config_group *group = to_config_group(item); + struct epf_ntb *ntb = to_epf_ntb(group); + u32 val; + int ret; + + ret = kstrtou32(page, 0, &val); + if (ret) + return ret; + + if (val > MAX_MW) + return -EINVAL; + + ntb->num_mws = val; + + return len; +} + +EPF_NTB_R(spad_count) +EPF_NTB_W(spad_count) +EPF_NTB_R(db_count) +EPF_NTB_W(db_count) +EPF_NTB_R(num_mws) +EPF_NTB_R(vbus_number) +EPF_NTB_W(vbus_number) +EPF_NTB_MW_R(mw1) +EPF_NTB_MW_W(mw1) +EPF_NTB_MW_R(mw2) +EPF_NTB_MW_W(mw2) +EPF_NTB_MW_R(mw3) +EPF_NTB_MW_W(mw3) +EPF_NTB_MW_R(mw4) +EPF_NTB_MW_W(mw4) + +CONFIGFS_ATTR(epf_ntb_, spad_count); +CONFIGFS_ATTR(epf_ntb_, db_count); +CONFIGFS_ATTR(epf_ntb_, num_mws); +CONFIGFS_ATTR(epf_ntb_, mw1); +CONFIGFS_ATTR(epf_ntb_, mw2); +CONFIGFS_ATTR(epf_ntb_, mw3); +CONFIGFS_ATTR(epf_ntb_, mw4); +CONFIGFS_ATTR(epf_ntb_, vbus_number); + +static struct configfs_attribute *epf_ntb_attrs[] = { + &epf_ntb_attr_spad_count, + &epf_ntb_attr_db_count, + &epf_ntb_attr_num_mws, + &epf_ntb_attr_mw1, + &epf_ntb_attr_mw2, + &epf_ntb_attr_mw3, + &epf_ntb_attr_mw4, + &epf_ntb_attr_vbus_number, + NULL, +}; + +static const struct config_item_type ntb_group_type = { + .ct_attrs = epf_ntb_attrs, + .ct_owner = THIS_MODULE, +}; + +/** + * epf_ntb_add_cfs() - Add configfs directory specific to NTB + * @epf: NTB endpoint function device + * @group: A pointer to the config_group structure referencing a group of + * config_items of a specific type that belong to a specific sub-system. + * + * Add configfs directory specific to NTB. This directory will hold + * NTB specific properties like db_count, spad_count, num_mws etc., + */ +static struct config_group *epf_ntb_add_cfs(struct pci_epf *epf, + struct config_group *group) +{ + struct epf_ntb *ntb = epf_get_drvdata(epf); + struct config_group *ntb_group = &ntb->group; + struct device *dev = &epf->dev; + + config_group_init_type_name(ntb_group, dev_name(dev), &ntb_group_type); + + return ntb_group; +} + +/*==== virtual PCI bus driver, which only load virutal ntb pci driver ====*/ + +#define VPCI_BUS_NUM 0x10 + +uint32_t pci_space[] = { + (VNTB_VID | (VNTB_PID << 16)), //DeviceID, Vendor ID + 0, // status, Command + 0xffffffff, // Class code, subclass, prog if, revision id + 0x40, //bist, header type, latency Timer, cache line size + 0, //bar 0 + 0, //bar 1 + 0, //bar 2 + 0, //bar 3 + 0, //bar 4 + 0, //bar 5 + 0, //cardbus cis point + 0, //Subsystem ID Subystem vendor id + 0, //ROM Base Address + 0, //Reserved, Cap. Point + 0, //Reserved, + 0, //Max Lat, Min Gnt, interrupt pin, interrupt line +}; + +int pci_read(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *val) +{ + if (devfn == 0) { + memcpy(val, ((uint8_t *)pci_space) + where, size); + return 0; + } + return -1; +} + +int pci_write(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 val) +{ + return 0; +} + +struct pci_ops vpci_ops = { + .read = pci_read, + .write = pci_write, +}; + +static int vpci_bus(void *sysdata) +{ + struct pci_bus *vpci_bus; + + vpci_bus = pci_scan_bus(VPCI_BUS_NUM, &vpci_ops, sysdata); + if (vpci_bus) + pr_err("create pci bus\n"); + + pci_bus_add_devices(vpci_bus); + + return 0; +} + +/*==================== Virtual PCIe NTB driver ==========================*/ + +static int vntb_epf_mw_count(struct ntb_dev *ntb, int pidx) +{ + struct epf_ntb *ndev = ntb_ndev(ntb); + + return ndev->num_mws; +} + +static int vntb_epf_spad_count(struct ntb_dev *ntb) +{ + return ntb_ndev(ntb)->spad_count; +} + +static int vntb_epf_peer_mw_count(struct ntb_dev *ntb) +{ + return ntb_ndev(ntb)->num_mws; +} + +static u64 vntb_epf_db_valid_mask(struct ntb_dev *ntb) +{ + return BIT_ULL(ntb_ndev(ntb)->db_count) - 1; +} + +static int vntb_epf_db_set_mask(struct ntb_dev *ntb, u64 db_bits) +{ + return 0; +} + +static int vntb_epf_mw_set_trans(struct ntb_dev *ndev, int pidx, int idx, + dma_addr_t addr, resource_size_t size) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + struct pci_epf_bar *epf_bar; + enum pci_barno barno; + int ret; + struct device *dev; + + dev = &ntb->ntb.dev; + barno = ntb->epf_ntb_bar[BAR_MW0 + idx]; + epf_bar = &ntb->epf->bar[barno]; + epf_bar->phys_addr = addr; + epf_bar->barno = barno; + epf_bar->size = size; + + ret = pci_epc_set_bar(ntb->epf->epc, 0, 0, epf_bar); + if (ret) { + dev_err(dev, "failure set mw trans\n"); + return ret; + } + return 0; +} + +static int vntb_epf_mw_clear_trans(struct ntb_dev *ntb, int pidx, int idx) +{ + return 0; +} + +static int vntb_epf_peer_mw_get_addr(struct ntb_dev *ndev, int idx, + phys_addr_t *base, resource_size_t *size) +{ + + struct epf_ntb *ntb = ntb_ndev(ndev); + + if (base) + *base = ntb->vpci_mw_phy[idx]; + + if (size) + *size = ntb->mws_size[idx]; + + return 0; +} + +static int vntb_epf_link_enable(struct ntb_dev *ntb, + enum ntb_speed max_speed, + enum ntb_width max_width) +{ + return 0; +} + +static u32 vntb_epf_spad_read(struct ntb_dev *ndev, int idx) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + int off = ntb->reg->spad_offset, ct = ntb->reg->spad_count * 4; + u32 val; + void __iomem *base = ntb->reg; + + val = readl(base + off + ct + idx * 4); + return val; +} + +static int vntb_epf_spad_write(struct ntb_dev *ndev, int idx, u32 val) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + struct epf_ntb_ctrl *ctrl = ntb->reg; + int off = ctrl->spad_offset, ct = ctrl->spad_count * 4; + void __iomem *base = ntb->reg; + + writel(val, base + off + ct + idx * 4); + return 0; +} + +static u32 vntb_epf_peer_spad_read(struct ntb_dev *ndev, int pidx, int idx) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + struct epf_ntb_ctrl *ctrl = ntb->reg; + int off = ctrl->spad_offset; + void __iomem *base = ntb->reg; + u32 val; + + val = readl(base + off + idx * 4); + return val; +} + +static int vntb_epf_peer_spad_write(struct ntb_dev *ndev, int pidx, int idx, u32 val) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + struct epf_ntb_ctrl *ctrl = ntb->reg; + int off = ctrl->spad_offset; + void __iomem *base = ntb->reg; + + writel(val, base + off + idx * 4); + return 0; +} + +static int vntb_epf_peer_db_set(struct ntb_dev *ndev, u64 db_bits) +{ + u32 interrupt_num = ffs(db_bits) + 1; + struct epf_ntb *ntb = ntb_ndev(ndev); + u8 func_no, vfunc_no; + int ret; + + func_no = ntb->epf->func_no; + vfunc_no = ntb->epf->vfunc_no; + + ret = pci_epc_raise_irq(ntb->epf->epc, + func_no, + vfunc_no, + PCI_EPC_IRQ_MSI, + interrupt_num + 1); + if (ret) { + dev_err(&ntb->ntb.dev, "intf: Failed to raise IRQ\n"); + return ret; + } + + return 0; +} + +static u64 vntb_epf_db_read(struct ntb_dev *ndev) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + + return ntb->db; +} + +static int vntb_epf_mw_get_align(struct ntb_dev *ndev, int pidx, int idx, + resource_size_t *addr_align, + resource_size_t *size_align, + resource_size_t *size_max) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + + if (addr_align) + *addr_align = SZ_4K; + + if (size_align) + *size_align = 1; + + if (size_max) + *size_max = ntb->mws_size[idx]; + + return 0; +} + +static u64 vntb_epf_link_is_up(struct ntb_dev *ndev, + enum ntb_speed *speed, + enum ntb_width *width) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + + return ntb->reg->link_status; +} + +static int vntb_epf_db_clear_mask(struct ntb_dev *ndev, u64 db_bits) +{ + return 0; +} + +static int vntb_epf_db_clear(struct ntb_dev *ndev, u64 db_bits) +{ + struct epf_ntb *ntb = ntb_ndev(ndev); + + ntb->db &= ~db_bits; + return 0; +} + +static int vntb_epf_link_disable(struct ntb_dev *ntb) +{ + return 0; +} + +static const struct ntb_dev_ops vntb_epf_ops = { + .mw_count = vntb_epf_mw_count, + .spad_count = vntb_epf_spad_count, + .peer_mw_count = vntb_epf_peer_mw_count, + .db_valid_mask = vntb_epf_db_valid_mask, + .db_set_mask = vntb_epf_db_set_mask, + .mw_set_trans = vntb_epf_mw_set_trans, + .mw_clear_trans = vntb_epf_mw_clear_trans, + .peer_mw_get_addr = vntb_epf_peer_mw_get_addr, + .link_enable = vntb_epf_link_enable, + .spad_read = vntb_epf_spad_read, + .spad_write = vntb_epf_spad_write, + .peer_spad_read = vntb_epf_peer_spad_read, + .peer_spad_write = vntb_epf_peer_spad_write, + .peer_db_set = vntb_epf_peer_db_set, + .db_read = vntb_epf_db_read, + .mw_get_align = vntb_epf_mw_get_align, + .link_is_up = vntb_epf_link_is_up, + .db_clear_mask = vntb_epf_db_clear_mask, + .db_clear = vntb_epf_db_clear, + .link_disable = vntb_epf_link_disable, +}; + +static int pci_vntb_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + int ret; + struct epf_ntb *ndev = (struct epf_ntb *)pdev->sysdata; + struct device *dev = &pdev->dev; + + ndev->ntb.pdev = pdev; + ndev->ntb.topo = NTB_TOPO_NONE; + ndev->ntb.ops = &vntb_epf_ops; + + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); + if (ret) { + dev_err(dev, "Cannot set DMA mask\n"); + return -1; + } + + ret = ntb_register_device(&ndev->ntb); + if (ret) { + dev_err(dev, "Failed to register NTB device\n"); + goto err_register_dev; + } + + dev_info(dev, "PCI Virtual NTB driver loaded\n"); + return 0; + +err_register_dev: + return -1; +} + +static const struct pci_device_id pci_vntb_table[] = { + { + PCI_DEVICE(VNTB_VID, VNTB_PID), + }, + {}, +}; + +static struct pci_driver vntb_pci_driver = { + .name = "pci-vntb", + .id_table = pci_vntb_table, + .probe = pci_vntb_probe, +}; + +/* ============ PCIe EPF Driver Bind ====================*/ + +/** + * epf_ntb_bind() - Initialize endpoint controller to provide NTB functionality + * @epf: NTB endpoint function device + * + * Initialize both the endpoint controllers associated with NTB function device. + * Invoked when a primary interface or secondary interface is bound to EPC + * device. This function will succeed only when EPC is bound to both the + * interfaces. + */ +static int epf_ntb_bind(struct pci_epf *epf) +{ + struct epf_ntb *ntb = epf_get_drvdata(epf); + struct device *dev = &epf->dev; + int ret; + + if (!epf->epc) { + dev_dbg(dev, "PRIMARY EPC interface not yet bound\n"); + return 0; + } + + ret = epf_ntb_init_epc_bar(ntb); + if (ret) { + dev_err(dev, "Failed to create NTB EPC\n"); + goto err_bar_init; + } + + ret = epf_ntb_config_spad_bar_alloc(ntb); + if (ret) { + dev_err(dev, "Failed to allocate BAR memory\n"); + goto err_bar_alloc; + } + + ret = epf_ntb_epc_init(ntb); + if (ret) { + dev_err(dev, "Failed to initialize EPC\n"); + goto err_bar_alloc; + } + + epf_set_drvdata(epf, ntb); + + if (pci_register_driver(&vntb_pci_driver)) { + dev_err(dev, "failure register vntb pci driver\n"); + goto err_bar_alloc; + } + + vpci_bus(ntb); + + return 0; + +err_bar_alloc: + epf_ntb_config_spad_bar_free(ntb); + +err_bar_init: + epf_ntb_epc_destroy(ntb); + + return ret; +} + +/** + * epf_ntb_unbind() - Cleanup the initialization from epf_ntb_bind() + * @epf: NTB endpoint function device + * + * Cleanup the initialization from epf_ntb_bind() + */ +static void epf_ntb_unbind(struct pci_epf *epf) +{ + struct epf_ntb *ntb = epf_get_drvdata(epf); + + epf_ntb_epc_cleanup(ntb); + epf_ntb_config_spad_bar_free(ntb); + epf_ntb_epc_destroy(ntb); + + pci_unregister_driver(&vntb_pci_driver); +} + +// EPF driver probe +static struct pci_epf_ops epf_ntb_ops = { + .bind = epf_ntb_bind, + .unbind = epf_ntb_unbind, + .add_cfs = epf_ntb_add_cfs, +}; + +/** + * epf_ntb_probe() - Probe NTB function driver + * @epf: NTB endpoint function device + * + * Probe NTB function driver when endpoint function bus detects a NTB + * endpoint function. + */ +static int epf_ntb_probe(struct pci_epf *epf) +{ + struct epf_ntb *ntb; + struct device *dev; + + dev = &epf->dev; + + ntb = devm_kzalloc(dev, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + + epf->header = &epf_ntb_header; + ntb->epf = epf; + epf_set_drvdata(epf, ntb); + + dev_info(dev, "pci-ep epf driver loaded\n"); + return 0; +} + +static const struct pci_epf_device_id epf_ntb_ids[] = { + { + .name = "pci_epf_vntb", + }, + {}, +}; + +static struct pci_epf_driver epf_ntb_driver = { + .driver.name = "pci_epf_vntb", + .probe = epf_ntb_probe, + .id_table = epf_ntb_ids, + .ops = &epf_ntb_ops, + .owner = THIS_MODULE, +}; + + +static int __init epf_ntb_init(void) +{ + int ret; + + kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM | + WQ_HIGHPRI, 0); + ret = pci_epf_register_driver(&epf_ntb_driver); + if (ret) { + destroy_workqueue(kpcintb_workqueue); + pr_err("Failed to register pci epf ntb driver --> %d\n", ret); + return ret; + } + + return 0; +} +module_init(epf_ntb_init); + +static void __exit epf_ntb_exit(void) +{ + pci_epf_unregister_driver(&epf_ntb_driver); + destroy_workqueue(kpcintb_workqueue); +} +module_exit(epf_ntb_exit); + +MODULE_DESCRIPTION("PCI EPF NTB DRIVER"); +MODULE_AUTHOR("Frank Li "); +MODULE_LICENSE("GPL v2"); From patchwork Tue Feb 15 05:38:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Li X-Patchwork-Id: 12746516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C965BC433EF for ; Tue, 15 Feb 2022 05:39:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233097AbiBOFja (ORCPT ); Tue, 15 Feb 2022 00:39:30 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:40892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234123AbiBOFj1 (ORCPT ); Tue, 15 Feb 2022 00:39:27 -0500 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10085.outbound.protection.outlook.com [40.107.1.85]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 029E11EC49 for ; Mon, 14 Feb 2022 21:39:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l7byLbslgnDvhzhCY6AxAPvUj5a3P0p/41UpDE9IitZXxf7oucsboHwo4EhSePX0H2u2jWBED3znGXmk32KsJUDxgyfxf8J9KPOly5/qGykIzeLa0CWhb/1PtYArtyt5GK/GlR9rh7yDNF/aE0/qAAwNy6nz9YeRN0hk8I94sZuD4LHvjCT8rRlWafddVTBOTCpf2vqaO+Op56m0DRav88JrfZPUe49NrsQFHMNHYhqtN5Y+kCKkMCHur5K40AtjR06v+5bBNI+4tLraHfcRmOXsdd2RPIFUtzwBo9IfSydbVv+Ehh7ih3atPTt5RujOTFDDjNaTRSpDmqBYWaJApw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SiOiY5ce5NC7bLWyHbPSmLTdsW2PGxGYYR9XE4+Xa2M=; b=h8NtzuPdLxLoR6MYeGvhEbikfgOJICexP5RfKO3Fi0u6fbwBH9XmUbaCesY7nFYGhsIlNMCmNiIf6YiMWMVzPAftOv81G/KTItCSgL0Er9c7bTSwoI9kCMXZ3ID4lRl3OlTzsyqIFDgUspuuQphqUFuwaLX50jeqxj3OsdCl7N5dGVx9WDKIIOSO0D+P9+Gh42LN6qoqRJ63Zq4nuhXg7wE5rVPH9iPYpHCYlJGnKMRcKInqPyfTWeu1voAKyZ76Fkj/Y1P8iOBzy7ittQVOnR7cGkoH5+9aMym/lCMuXDDmOWgOSB9FIJjugzAXfRx6uFm2ZxcTrYaht9m2B0uaNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SiOiY5ce5NC7bLWyHbPSmLTdsW2PGxGYYR9XE4+Xa2M=; b=S2wHtkKvmWAvWYSLavhGTVhYQknUolpT7ec1h6+LI3e57BwR6SZOKnwuItsodvFv45dW4DFHEP3q23Rl6yl87k5RxvfyRwwiL7uISoE86hDaiwlVmDTeDlJnUVo8uanXh+cCOKRKoRcEETXSBsSIDX0hn/g24PZCXT2B0L4kxFs= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nxp.com; Received: from PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) by VI1PR04MB2976.eurprd04.prod.outlook.com (2603:10a6:802:8::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.18; Tue, 15 Feb 2022 05:39:13 +0000 Received: from PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e]) by PAXPR04MB9186.eurprd04.prod.outlook.com ([fe80::4881:43a4:4127:f25e%6]) with mapi id 15.20.4975.018; Tue, 15 Feb 2022 05:39:13 +0000 From: Frank Li To: linux-pci@vger.kernel.org, kishon@ti.com, lorenzo.pieralisi@arm.com, kw@linux.com, jingoohan1@gmail.com, gustavo.pimentel@synopsys.com, lznuaa@gmail.com, hongxing.zhu@nxp.com Subject: [RFC PATCH 4/4] Documentation: PCI: Add specification for the PCI vNTB function device Date: Mon, 14 Feb 2022 23:38:44 -0600 Message-Id: <20220215053844.7119-5-Frank.Li@nxp.com> X-Mailer: git-send-email 2.24.0.rc1 In-Reply-To: <20220215053844.7119-1-Frank.Li@nxp.com> References: <20220215053844.7119-1-Frank.Li@nxp.com> X-ClientProxiedBy: BYAPR21CA0007.namprd21.prod.outlook.com (2603:10b6:a03:114::17) To PAXPR04MB9186.eurprd04.prod.outlook.com (2603:10a6:102:232::18) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae010d90-5ff7-4cf4-a801-08d9f0457f76 X-MS-TrafficTypeDiagnostic: VI1PR04MB2976:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: erT1LsaVv6CEt/aC/7FzMROCMyOk39AX4Dc/KCrQjp5xnsUXe6OjfcbBX7rEKyE2k19vna1WEElXV8rECf77Rq8S1zHbtQ0P743oiF7UWYzJNsal5C8twVG0gSsazwmo74OUinEs99Uc8yzKAALHbh6GCfwqKdWLzCiyRAIZk6U88ZLFb8rKqhI1NO+kDRfOO6MYR9S5Ja/gvaCgOVdN9p5gGLchCKjQtan/SBvJPmDPRKyvQ2i/IOJCzyXQpm/1+v6WEbnd+d9QGzFwa/k7VH8ccUuhVVdljHYqa4H8lf2+o6Lnh0t1aF31Px/qHDyQUfa2L6B+1hdhy97Hh+0Q5zt3dKQlv+PqoBO32rbV2jtawnzzFy4Wg9Lj0oLUOjxG5/Zc+8D0N7UlJO7ZNhxU/DLyq4mA2jYcwhQ9yzK9OP7cS0BWm1Xr3jPeezVmlL1BQ8vfbR/mqLPbxBUEMtUr8C7flNYPPxRGXfeVB6JgXt8ZTs/xvkPwhtfMyryNIUJPQDyFTaZlKcSy9bfiybDtG5ZDgTYCseiKP8jtct5nVTUhO961C1lOLYFkR4sgo23Y+L4ueiBkEHRynKJxjgKhFXn4yhPwPd7Ti9nYzyjTgjGc4XBp5eWqU4jmyDWxEjgNtoAmeiP0E67+FgpXfjwx/vg1cRUXXMMZmMxuNQg42oGCCFW/1XATPKNWT8LjWkxTag4E250yVp2sK8XR3odDuw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR04MB9186.eurprd04.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(6506007)(5660300002)(26005)(52116002)(38350700002)(186003)(6512007)(83380400001)(38100700002)(86362001)(36756003)(2906002)(8936002)(8676002)(508600001)(66476007)(316002)(2616005)(66556008)(66946007)(6666004)(1076003)(6486002)(6636002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PwyMXfu1YZyCIxZPzLHjiMeVpXdv1gdS2E79TvNBifcTc3c37dGnsjX6kqmyEc/s1f96KTyPqpNVyKBxvRhrsS7xSxTY/AWhZnmVuyR5l0jUESpIA8Hbfa/e6gjBnatJTbNbJK2bLgX/cDpJKR6dk4wqGRuF8WdgJdksyFeyZVNEoG8VDblplgk6XQvpfj1FmIgY1bkFjgHjozhbwHZwDEeyCnMGrCh6UTky0UtYK7TkY+CDVevPSwniot96qc+BsLDcEUmIzNEEf94mqnMcjh8RAmnXlb6yO+IeKyPfdmrwGah0/WywO4Kytf72cVg6I7sAFo1BAliLr35fELF8i0QPI5aoeslEjFEPcVnsUe3jDJ8r4q4tN8c6Tx7jaYAaIOq+OTVIO2RKTPWf3GM/AqBQ6DrTBohjKRkW8vB1Yk9okDonkg/vgR/+cDaY9JVrZIkwIVayV0EPMgIYXuSVo/iQqswq5BjHfIETGnBFQK1RvGVnLIUgwHHcaWUmaJXJEkAH2QH9ZnIbH2HBAWDJ/ezLkCUyRp5pR0vO2uXkVrlNg5Uq7FECsek35fE/zD4u4FozIpZReu0LypROIWBpO6yI7c2LYExe/wd3g+apjx3tXjp8QccTPhrq/g7p/uM5JWH18uMmXIclJyW6Vl0rR7gpxTNmKjz2JsRLoOcr9EbLxAw7be5KAZF7XW3L7/4QCWvZFS9KrM9TxR8dA8v798yzZL/WS7eU7tNp8aWN/vjx/GmIJsvxF9FHlEszyuG9AhJkoxgsxmRnh14tREuqt4XnLNVFQUNglhmXJJOI2yM/rCSnJmSDivX+IxF6O7eYQRRM5rmm9O/02qYbqf+spgEOcMiJeGYXai3Zp38F2KpoYdiG5dp7CAgNvhFuGvpdgzL2nF0f8I1nuyO90POY3fawz6o753S/iUX/b9lJzV7mqGMldYsPYhU58q+o/ABia2uhEpkHv2ktxIYTKLyItrwmrGZCTUMlWBU0+HtwaGmNmSG5TOAScOmxg4fzc8zorRV5RTRdiJ2d94or78HSn5OLkg8L51ciFYzpvDb0hLbmVSbnFJcDlgSTYj99uRu9uQKB22AZ3OxBcF4XvrKkna4llpN6PumhdwHmtlCqh6uVUdN9/pxPgFjMf91Ci/ycejl9Z1H7A3sKrFH7fuvhrrgzTQ+Emt18TfGsgkfmBmp6k/g3PHbTNIFO/DRfKmiopGIewgDY7Sf3KFANzw3wvWJeXmI/bIOhG/EIpomVuhgJ3KR15+0ZIgDjElDn/5STHTZTpiY56SB7dnWF1Jd9KlboMZDZmly3vuYMTareFQOPQu6CDHDA/CuLZhCVkuGXufcdgswReTg3YY2OupetiDtLXnEe5n7ZkaCKLt8B5uSxeCn6HcwwfKFs5Z2NO7BN2HIU8pDHnvcklC1nWb/wvtvx22bOY+6RaMFtJxggkoweyMPP/VfHO8JKxRRL3OreqRce+bedwB9pD8zHAht3fteAGIFb2OL43jvIVxnYTa6FU5P6Pla9dWr77MF80oK5bTaOkdh1zE2gCC7ttMthO7fPUZeLx6vw1jCRftEzcy26fqRqXNHycAz6J5Fjt5A3P2L9sA7gvyGJ2mSwmJZJmcSm1IxFy9hIm/HkMNvbUAs= X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae010d90-5ff7-4cf4-a801-08d9f0457f76 X-MS-Exchange-CrossTenant-AuthSource: PAXPR04MB9186.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Feb 2022 05:39:13.4496 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2unBSMm3BEfLNATqhoNaEZAY3pxNkDURyUs6qd3cr2sHkVx8GO3B/i/AXy2t1rZoG1n6hPWsWy3PoIthsoKzUg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB2976 Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add specification for the PCI vNTB function device. The endpoint function driver and the host PCI driver should be created based on this specification. Signed-off-by: Frank Li --- Documentation/PCI/endpoint/index.rst | 2 + .../PCI/endpoint/pci-vntb-function.rst | 126 ++++++++++++++ Documentation/PCI/endpoint/pci-vntb-howto.rst | 161 ++++++++++++++++++ 3 files changed, 289 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-vntb-function.rst create mode 100644 Documentation/PCI/endpoint/pci-vntb-howto.rst diff --git a/Documentation/PCI/endpoint/index.rst b/Documentation/PCI/endpoint/index.rst index 38ea1f604b6d3..4d2333e7ae067 100644 --- a/Documentation/PCI/endpoint/index.rst +++ b/Documentation/PCI/endpoint/index.rst @@ -13,6 +13,8 @@ PCI Endpoint Framework pci-test-howto pci-ntb-function pci-ntb-howto + pci-vntb-function + pci-vntb-howto function/binding/pci-test function/binding/pci-ntb diff --git a/Documentation/PCI/endpoint/pci-vntb-function.rst b/Documentation/PCI/endpoint/pci-vntb-function.rst new file mode 100644 index 0000000000000..cad8013e88390 --- /dev/null +++ b/Documentation/PCI/endpoint/pci-vntb-function.rst @@ -0,0 +1,126 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI vNTB Function +================= + +:Author: Frank Li + +The difference between PCI NTB function and PCI vNTB function is + +PCI NTB function need at two endpoint instances and connect HOST1 +and HOST2. + +PCI vNTB function only use one host and one endpoint(EP), use NTB +connect EP and PCI host + +.. code-block:: text + + + +------------+ +---------------------------------------+ + | | | | + +------------+ | +--------------+ + | NTB | | | NTB | + | NetDev | | | NetDev | + +------------+ | +--------------+ + | NTB | | | NTB | + | Transfer | | | Transfer | + +------------+ | +--------------+ + | | | | | + | PCI NTB | | | | + | EPF | | | | + | Driver | | | PCI Virtual | + | | +---------------+ | NTB Driver | + | | | PCI EP NTB |<------>| | + | | | FN Driver | | | + +------------+ +---------------+ +--------------+ + | | | | | | + | PCI BUS | <-----> | PCI EP BUS | | Virtual PCI | + | | PCI | | | BUS | + +------------+ +---------------+--------+--------------+ + PCI RC PCI EP + +Constructs used for Implementing vNTB +===================================== + + 1) Config Region + 2) Self Scratchpad Registers + 3) Peer Scratchpad Registers + 4) Doorbell (DB) Registers + 5) Memory Window (MW) + + +Config Region: +-------------- + +It is same as PCI NTB Function driver + +Scratchpad Registers: +--------------------- + + It is appended after Config region. + + +--------------------------------------------------+ Base + | | + | | + | | + | Common Config Register | + | | + | | + | | + +-----------------------+--------------------------+ Base + span_offset + | | | + | Peer Span Space | Span Space | + | | | + | | | + +-----------------------+--------------------------+ Base + span_offset + | | | + span_count * 4 + | | | + | Span Space | Peer Span Space | + | | | + +-----------------------+--------------------------+ + Virtual PCI Pcie Endpoint + NTB Driver NTB Driver + + +Doorbell Registers: +------------------- + + Doorbell Registers are used by the hosts to interrupt each other. + +Memory Window: +-------------- + + Actual transfer of data between the two hosts will happen using the + memory window. + +Modeling Constructs: +==================== + +32-bit BARs. + +====== =============== +BAR NO CONSTRUCTS USED +====== =============== +BAR0 Config Region +BAR1 Doorbell +BAR2 Memory Window 1 +BAR3 Memory Window 2 +BAR4 Memory Window 3 +BAR5 Memory Window 4 +====== =============== + +64-bit BARs. + +====== =============================== +BAR NO CONSTRUCTS USED +====== =============================== +BAR0 Config Region + Scratchpad +BAR1 +BAR2 Doorbell +BAR3 +BAR4 Memory Window 1 +BAR5 +====== =============================== + + diff --git a/Documentation/PCI/endpoint/pci-vntb-howto.rst b/Documentation/PCI/endpoint/pci-vntb-howto.rst new file mode 100644 index 0000000000000..b4a679144692a --- /dev/null +++ b/Documentation/PCI/endpoint/pci-vntb-howto.rst @@ -0,0 +1,161 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=================================================================== +PCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide +=================================================================== + +:Author: Frank Li + +This document is a guide to help users use pci-epf-vntb function driver +and ntb_hw_epf host driver for NTB functionality. The list of steps to +be followed in the host side and EP side is given below. For the hardware +configuration and internals of NTB using configurable endpoints see +Documentation/PCI/endpoint/pci-vntb-function.rst + +Endpoint Device +=============== + +Endpoint Controller Devices +--------------------------- + +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + 5f010000.pcie_ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + 5f010000.pcie_ep + +Endpoint Function Drivers +------------------------- + +To find the list of endpoint function drivers in the system:: + + # ls /sys/bus/pci-epf/drivers + pci_epf_ntb pci_epf_test pci_epf_vntb + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/functions + pci_epf_ntb pci_epf_test pci_epf_vntb + + +Creating pci-epf-vntb Device +---------------------------- + +PCI endpoint function device can be created using the configfs. To create +pci-epf-vntb device, the following commands can be used:: + + # mount -t configfs none /sys/kernel/config + # cd /sys/kernel/config/pci_ep/ + # mkdir functions/pci_epf_vntb/func1 + +The "mkdir func1" above creates the pci-epf-ntb function device that will +be probed by pci_epf_vntb driver. + +The PCI endpoint framework populates the directory with the following +configurable fields:: + + # ls functions/pci_epf_ntb/func1 + baseclass_code deviceid msi_interrupts pci-epf-ntb.0 + progif_code secondary subsys_id vendorid + cache_line_size interrupt_pin msix_interrupts primary + revid subclass_code subsys_vendor_id + +The PCI endpoint function driver populates these entries with default values +when the device is bound to the driver. The pci-epf-vntb driver populates +vendorid with 0xffff and interrupt_pin with 0x0001:: + + # cat functions/pci_epf_vntb/func1/vendorid + 0xffff + # cat functions/pci_epf_vntb/func1/interrupt_pin + 0x0001 + + +Configuring pci-epf-vntb Device +------------------------------- + +The user can configure the pci-epf-vntb device using its configfs entry. In order +to change the vendorid and the deviceid, the following +commands can be used:: + + # echo 0x1957 > functions/pci_epf_vntb/func1/vendorid + # echo 0x0809 > functions/pci_epf_vntb/func1/deviceid + +In order to configure NTB specific attributes, a new sub-directory to func1 +should be created:: + + # mkdir functions/pci_epf_vntb/func1/pci_epf_vntb.0/ + +The NTB function driver will populate this directory with various attributes +that can be configured by the user:: + + # ls functions/pci_epf_vntb/func1/pci_epf_vntb.0/ + db_count mw1 mw2 mw3 mw4 num_mws + spad_count + +A sample configuration for NTB function is given below:: + + # echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count + # echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count + # echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws + # echo 0x100000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1 + +Binding pci-epf-ntb Device to EP Controller +-------------------------------------------- + +NTB function device should be attached to PCI endpoint controllers +connected to the host. + + # ln -s controllers/5f010000.pcie_ep functions/pci-epf-ntb/func1/primary + +Once the above step is completed, the PCI endpoint controllers are ready to +establish a link with the host. + + +Start the Link +-------------- + +In order for the endpoint device to establish a link with the host, the _start_ +field should be populated with '1'. For NTB, both the PCI endpoint controllers +should establish link with the host (imx8 don't need this steps):: + + # echo 1 > controllers/5f010000.pcie_ep/start + +RootComplex Device +================== + +lspci Output at Host side +------------------------ + +Note that the devices listed here correspond to the values populated in +"Creating pci-epf-ntb Device" section above:: + + # lspci + 00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) + 01:00.0 RAM memory: Freescale Semiconductor Inc Device 0809 + +Endpoint Device / Virtual PCI bus +================================= + +lspci Output at EP Side / Virtual PCI bus +----------------------------------------- + +Note that the devices listed here correspond to the values populated in +"Creating pci-epf-ntb Device" section above:: + + # lspci + 10:00.0 Unassigned class [ffff]: Dawicontrol Computersysteme GmbH Device 1234 (rev ff) + +Using ntb_hw_epf Device +----------------------- + +The host side software follows the standard NTB software architecture in Linux. +All the existing client side NTB utilities like NTB Transport Client and NTB +Netdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB +function device. + +For more information on NTB see +:doc:`Non-Transparent Bridge <../../driver-api/ntb>`