From patchwork Fri Sep 20 22:34:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808718 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2052.outbound.protection.outlook.com [40.107.236.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 008BE18A6D3; Fri, 20 Sep 2024 22:35:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.52 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871714; cv=fail; b=Dp1YvspHygfbq0PgeQ+Pk7gStA1ae+/nIpBIeDPd61XdE//wycx0TRJpeyVmuW++MIxhB72+t4XE1Rh36/Yd9AES/e1GRcaKmG7MeTSMf7Q/gN6o39yjdRF9kzP9QgIiybsjs03BEWcxRPs9o72lqjXConFDMoUko0x5qsN13z0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871714; c=relaxed/simple; bh=3FJ+wEiysLCQ56DMR7pDImSavuwsQIQisuk1b47zCA0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=VxS3KkEzR0SHk8lTBfX3mWzxbL22NXo0EZ3lTiuHq/j1xmiTQv+jXW2dwEs0MQcNmtlM2RcFIlCCi728DAwnehNiCMcWzlVHEnMFVgo/v4BTpfdM+90W0axINmRIUtD4u7TayzFuJBFXbUCnmDzotzHJxyT1Zx51+oc0EYQuHE4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=KTfl7yNs; arc=fail smtp.client-ip=40.107.236.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="KTfl7yNs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=K4fCh37/Nc6JP4e1fFjkGumvK2M0B5GQPB2p073anS/XP0xBAzTsQI8HG1FIEdIKFK/mn8aqy6lZ/BEohIOJj1bqv1DnAm4Qf5uvI3Xg0GcF461vcDNZxhGW9ycPhY21aX6iCqlap4L6J1lzqHr9OTdBcsTWH0JEabCFIQxBP1FI+bm/J4QZUGCRuCMscI7rHr33i6bVJlz0ldsneHEizbO+yaU9Il4Y/+VAAEU/ytHX+oDKiaW1z0a0+N95TxNKjedeq4c0xlWMFC52B/tn3pieEhv9t4Q4iKnkK/rRat7tdEjHz2Iq8rBAfDY2nFH0hPdFkSRJteZLRhwgX5xkWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dgzxLMZN7EzbADKMTn3VwSAg5y6IGbA7pknUGYtRBKs=; b=Qn1nnWWrcSO3ijfa0tGodzrdmW48wfcjkgAF54jXeVunyB7rxQXC75konGABVK2taW6hD5b9n2KKaHSjoQ+mfBctH/SS1OwWMV6ismyhj/JE733OQAcR+BSZLN57BUlALyJ1s3V84oK33LUDfVhX6sfRrcokIXvtxEhVJT1nv7SKSUSoppdt1F9Jwy/sIPeq7NDUbhwkBUg+RwhpcAH6e8NrE479MPxFq3fT9Y6GDwkvKVZpPk+iOe9AhFAJDsKWURlgzRPniSEr9V8HS0tQjxx7JB9+39/3Q6oAuPZRsT5J6Lecmr1Ezj+H7Jqn3AiQD5KHKKGxfbTImsZ49wKtGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dgzxLMZN7EzbADKMTn3VwSAg5y6IGbA7pknUGYtRBKs=; b=KTfl7yNs2uA6AurjoDRvtJo0LEjOG642+23yzPeNBndqdAFN/YtmjGgsnRgR/H11y9BCxPNdW368BIYElCx+Y7yzqwqEHfAG97BZduVg2lTXqDyuSNlcSvrWHiLTS5vALPmRR32RNHdQu67TAX2TT//td3oGEDiHwGsyW6xan1ijcbFbVe9SVJGemqr/T3tbQeZxbDeGdahwq5dO6pyc+wISu2KT23LZu0QPHd6+MkTh7vQIRuCY8Ew8wPpmA1DfStIEZuMtQaPNSwsWddDRLYR+iIiURfnFxePIwNboXTfnm7cd9z0Gra0cuxIfYBsMUm+jgGBQuYOPa22GBmoLCA== Received: from SJ0PR13CA0210.namprd13.prod.outlook.com (2603:10b6:a03:2c3::35) by DM6PR12MB4138.namprd12.prod.outlook.com (2603:10b6:5:220::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.22; Fri, 20 Sep 2024 22:35:08 +0000 Received: from CY4PEPF0000E9D4.namprd03.prod.outlook.com (2603:10b6:a03:2c3:cafe::8b) by SJ0PR13CA0210.outlook.office365.com (2603:10b6:a03:2c3::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Fri, 20 Sep 2024 22:35:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D4.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:06 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:52 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:52 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:51 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 01/13] cxl: allow a type-2 device not to have memory device registers Date: Fri, 20 Sep 2024 15:34:34 -0700 Message-ID: <20240920223446.1908673-2-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D4:EE_|DM6PR12MB4138:EE_ X-MS-Office365-Filtering-Correlation-Id: 1ea9ad7c-e11e-48ce-84d3-08dcd9c47a4e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: 3XUeIwtZp6c8mDklwZSH8NSgId07HnE86ZMlV10zXgtEdFv92LqHT2w6JE4KWVD/efZ4Puk9zqgV2FdGlatxsWH57Wrygo0xeMYXnILaAyFIfPCJRLhGv36ud9eu8NEXV3CC+ClR0V80kW+pDcjcN6LJn1nNU5iV39aGnvhVDA2Fw9kpBy3YpLLtZ8QXwf/H7bXdQr1KrgU34X5I6a0E4SGU26JhPWCeouXXui19CXifVLN3FDJz3kweumIFiorIQ/DMZhW2MJrYBn9aIn9uUzitR5ED0U8kLz8xfr8YQDn5qCJPFWXZVDT0kOwO5S0avcsnb7eE7SzTUKIsYwCpInd8ZdKkjMTwMCIkShrfjUX/Y3NEXVry7xqryr/ZG46Sz8YVy8sDq2E4BF/x6kXBhAlAuuVYnCkDWl719QqQOSQFpo/UxP3E1fYilOGcWL3ga8BoHnwCS3oRhh6FsHH92g5mL8qo/NMTNq9crBS6CqVmuctlgXjmTW2pRaWyvO0x+WI7zd29+6q9YYt0PKLE0kxFk+Tqj6piXfpKcG6srmjcjHA7FYoep9jW6ByC+AadVtdX/wKJiVOrglOJR48pY8tx7dL+ZQnGNM4BnealeY+IKM8F+JMkuPjbbZCFsKRzTXoXq/FsXzw/T6QUbXYFqAKV+Olj+uRAegn4eRg197KQOeHZYs4/DXwuBLv1QSiRi3sqz4KHYR1Fo55you1YOW2T99+zeX2fbnpfF9Qmpu95BLYsMQ6SqRNQnlfh7f3JQljQ3Fn+74dt6CIiSqf+Zs7Y+YjnaQfy/8JU0YlnTLMlKGPmANQkqFZ2vRg/4FA5lk8ZTZXAI3HjiAbxEh4eloAcWi/ZSAeDVNCX07EVcPVkPmEmRkY6wi+BeLPmmX2+C4oP7hVYdbAxKcroTVl46oQ2y8h83vK8PHHg9xOq2iNvLK3DzVNRA1HJ02Zuvr4KJ2z3Ah6hjV6SaWuDh3+6+4VpxuOY8Ke1uryWPnZi/WbJSk4aKZfuNZQxePYcfiCAUlq2poBAGD+2mFnU0vW13HO3929AYHPTHqElKWaCJMQMz7PrRXT8N87A/nDVS3b3GhoCUpCcq37RqUYEZFmvzypccdYIeMN5cY/A55W1DRd8cw9XlGJDfvcSvLJIQeYpiN5XtlVkyWWvWeD/tfUTKu8Mdk/Ul8t1K2mr/1aiGTPRHo6S6jqe88XyRB6RDVb1wdPhGzsIhRE2p7/S/qrjNz12IfUpI+tJEQiet/085/KpqQ6Mf2TlqLjJLL7FIu8c7wgd/fmk6huEi+uQvtV0pTf03s7OZmHR9PQHANoMfjVv+T3auFgggIO+UHJHUACIUi1hHiBynMYhX2vktk6nnPdlZyiTVzU4bpLJUYCce0uza9QUuN7r4F/xrq7hA/Bs X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:06.7613 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1ea9ad7c-e11e-48ce-84d3-08dcd9c47a4e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4138 CXL memory device registers provide additional information about device memory and advanced control interface for type-3 device. However, it is not mandatory for a type-2 device. A type-2 device can have HDMs but not CXL memory device registers. Allow a type-2 device not to hanve memory device register when probing CXL registers. Signed-off-by: Zhi Wang --- drivers/cxl/pci.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index e00ce7f4d0f9..3fbee31995f1 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -529,13 +529,13 @@ int cxl_pci_accel_setup_regs(struct pci_dev *pdev, struct cxl_dev_state *cxlds) int rc; rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map, - cxlds->capabilities); - if (rc) - return rc; - - rc = cxl_map_device_regs(&map, &cxlds->regs.device_regs); - if (rc) - return rc; + cxlds->capabilities); + if (!rc) { + rc = cxl_map_device_regs(&map, &cxlds->regs.device_regs); + if (rc) + dev_dbg(&pdev->dev, + "Failed to map device registers.\n"); + } rc = cxl_pci_setup_regs(pdev, CXL_REGLOC_RBI_COMPONENT, &cxlds->reg_map, cxlds->capabilities); From patchwork Fri Sep 20 22:34:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808717 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2046.outbound.protection.outlook.com [40.107.223.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B868F178383; Fri, 20 Sep 2024 22:35:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871714; cv=fail; b=fPp8GUjCzUrGTcMQ+z/T9NJZkG3sliwnV8RYb0NH5XXJaXSXy7iMq8LTo59pRCWD2lHrSHcAZzI9X5E6CH41t0lfyV88+hpJ85Bu/ApbVWhHp4v4tN6sXwqfiHQFUlXw8JkHNLyij/1K8pLCCG7ttMtW6M3nMvw5fQfr3pbAEMY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871714; c=relaxed/simple; bh=fIEQ3gPMHlD88DIhTUaicLUwoVolyTAWkQLHwTTl6bg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a61gjbpY3OKIL6EYWsCt+gaaHGG/mqkONvCQ4Lj7SNxXcaHb8deuKgwWuHA2nJTliswiof07Jv8MjmwueuaZEL+W8R9RVFnMCZSjOaxpESZbU5FjwXpVU1jgKF/L4cw00DOTKmQzGOCiritpFrkvqzLtucSrK8X2VugdarQwqHU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=RWSLPszL; arc=fail smtp.client-ip=40.107.223.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="RWSLPszL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jsNjNLtIopKqD/ij5uChR72U7HZnEGdPE8dFfMMB1Ca01JuEwvC+DDhuu1+gjzlnGR8meY2/FhQXhYq1cABGC4UYk6TQubo0cOCKaJFp0MxOkLKuuFenRkt35NVlrbwnLv89hwUI+CzkcZnvo1KyUAdJUOqd7UQ6njHU6bsMznfyZUrX1ioD2XBaPnjepHTZbZ+HNem1/IDnn0rwWR7fpavBCSyPI4260ItBb/iW/hyR7T/YOCazh8hlXvm1O1xoHHS+hwuqP12Gz7uIMg3XSJq/vKgPIEKWw+mSAw91dNDvuEWHo45Ws+wIzB5shwXUUdR/KXHRlg02DGuYMxpb1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=p/DrpFWRD9dbbu+1OcVGRu3xpbNDnY2GDRIR1jRTWzo=; b=pFucRdkLrvjdwvNxeFurDCTgJ5CAyqrYQN3FLlpEUhxGdduwtp7icHFxDIOAgyaIeT5xwj96R5/QIZr4kdBzgiGMpQz/BRm00VCP9l5YoKk50U38eIJiypSjvsSMoXrZd2d3S9rMMCrXJ5gGCcyeNWoCMTqXxqjxB5xcDCZmZDStbJ2xLW+bqXEKKapThcE3I4uo06E0nzgh46q0lBDpcXL2mv+3ijk10qsek29IUlnJfIqN4dl0cBTqitAYzq/hmBgucqBloK782VCfvUbUDM9okr9vZ2sC8KbMwVUmHcKokgEvrlh6A8G49D4r+HZshqMr1WJ66NJ4QbHtEtceVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=p/DrpFWRD9dbbu+1OcVGRu3xpbNDnY2GDRIR1jRTWzo=; b=RWSLPszL1CY2naqN9ysioePpPFJbOAUO4GTrXQpdkpr8+t4F3yK21SjMCh8FmwBljoHWAlvOo6aW6AK5gTiIwk6daw5X5I8wwb874bHhuhj5LK787DH/GLXaNQfnu3fDdV5zIU1Muri9EMJqRdJ2AcQarMgCoriwUvXsEUSYAnxsI/aURuJvXuXwaOf6epqtFOxqFnKA8MVLlp0PEzbE8yrwxBzdmDb+XuEnYY3MlYI40YuaqRcBz21Rpkvdk+arWNI/J1IykBDa/+wy2VeZpQnDZgbVgiMvg93eyPG99eb5gZNGqK/EEptxttZHUkJhB/MIgf14eETuVcWmHltA9g== Received: from PH7PR10CA0006.namprd10.prod.outlook.com (2603:10b6:510:23d::25) by PH8PR12MB6817.namprd12.prod.outlook.com (2603:10b6:510:1c8::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.21; Fri, 20 Sep 2024 22:35:08 +0000 Received: from CY4PEPF0000E9D0.namprd03.prod.outlook.com (2603:10b6:510:23d:cafe::c2) by PH7PR10CA0006.outlook.office365.com (2603:10b6:510:23d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.30 via Frontend Transport; Fri, 20 Sep 2024 22:35:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D0.mail.protection.outlook.com (10.167.241.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:08 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:54 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:53 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:52 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 02/13] cxl: introduce cxl_get_hdm_info() Date: Fri, 20 Sep 2024 15:34:35 -0700 Message-ID: <20240920223446.1908673-3-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D0:EE_|PH8PR12MB6817:EE_ X-MS-Office365-Filtering-Correlation-Id: 6ea38c5f-0a95-4e0f-fd43-08dcd9c47b41 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|7416014|376014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: TPrJNDXkePvuaTF2/wTYvuQIkKOecXYxExRsJlD0lGbqt1zlQhkgCeJOVXZeRIL1TZHQ54oahaoIfkrruGwXXcQx7RAlDBQbsvw06qRdi5Q3MXP4OxHjKnUFjO8h9ii03PA1qRVoRvS0IxP3pv4PwhG8lqCO8I++iziaNgm2QGS1+tpB1FoAYcPEn6BJ7pE09+PMWcq5AXijDJ/4TsG2O7e5juYc8pqPhNo91fLLVK9F8ZgfC3iCt5cGWZ6oNPGUq0pfT/IH1DTu4zg4GzFGjcHO1kP8SM0SAg9GO6j4WBU4jDoQrIq0RvWQduC9GrFI4M1CvzYJiQJMvvfJ5J2HLZEbV3VaWNATJI0mlYq05wPTtGH0R13xNtd5qpoZrN2O/QZjIRgDK/TuMcE3HKXB3GIxrG/ezPpYCM1I1LHTKeGW77B0Dtke6kGzuOIxm46izjgoZhdmn32UXr3c7DSCChpgvLLtWGF1/vx4IbW4uprucRblReGXnsqpbKKvoooYi2R3ba3fYMbAbc66EYQSYWUpRZ+8VPftiyO1TKDjDATyq16zFzg0NVuM0IjRdixCIipzs+ockuzfe4ZJie1/K4lvtJSKK6JOptz1jfL2qpPTnh78lqSK1EZXUo9k2Hc5k08S6k4W4hVThzg9FEtcsIF0r4uf947GgrkBzszmM+xVo/e8AnkqKMfnVah3O0CHfO/wX3Q9iisLG8yMKiO9yVg5AjPyLcyK2kj+FG2cDRfV5u4sfNFKYYdo4mpPX+vtKjKhPT5T8vxSJgfu4+Pb5dViq6dO8H2X5li4p+NSPlE/3dQvKQw+625l80qtCQdXYY5XQve6CJL398vjSkZM5IaTCHjfkeYYnpXOxAaR9aLjbjdYp3CGvBXyz2AR8Xas7kUcDpzyfMRVtGIv1+ldlKUyEamzKz/XRKzXxoDCAYT5Xqzo4c2zes8mntXv3aDHqmCVWwUt+JvKcla5IEuSFsog8q3NS8Gt77xoOOs5Rq3tXLkLBvfBgqDDmG4rFJMNiIE6A2iILEFvhfxv4JSn+xhnNp7QtwlhF8FCwVx5e7zqqZZ5eRyNfavOyHI+eg9CyjNVU3wP2Y41Jy2lb/TXD/wubyf5veQxQVpfOE7k7rC41k31XTo4oWY/VcgspniDB0QReT+WJ0GUHPYcerwnGfkicXaqkkcmaQ0ZKpnEP80NBaJbmXzkttQvQhB6bUOpWY9QOElxprSCL1y0gENmmYRbi4SLwaXCYa2CV0nKU+THR7zRodtY6PlM0haZbC+zUFUWy+euajGzVZi6zt0QB0LzRQjGbNpZ8clgzgHL3BkPfzQi7aeJhtRVKafW4PdoU7zHDXVaTn1gocCrmOIaTarw8Wuc8+JYfUsT4P92rOenUi885o791kvof3bEHkOycgV3a4paScyNFxbE9S+Hg/t/YCaVneEVWl3kyaYuaJXSGYoZ4qdfFxc99K8oxHDz X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(7416014)(376014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:08.3726 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6ea38c5f-0a95-4e0f-fd43-08dcd9c47b41 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D0.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6817 CXL core has the information of what CXL register groups a device has. When initializing the device, the CXL core probes the register groups and saves the information. The probing sequence is quite complicated. vfio-cxl requires the HDM register information to emualte the HDM decoder registers. Introduce cxl_get_hdm_info() for vfio-cxl to leverage the HDM register information in the CXL core. Thus, it doesn't need to implement its own probing sequence. Signed-off-by: Zhi Wang --- drivers/cxl/core/pci.c | 28 ++++++++++++++++++++++++++++ drivers/cxl/cxlpci.h | 3 +++ include/linux/cxl_accel_mem.h | 2 ++ 3 files changed, 33 insertions(+) diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index a663e7566c48..7b6c2b6211b3 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -502,6 +502,34 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm, } EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, CXL); +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u32 *hdm_count, + u64 *hdm_reg_offset, u64 *hdm_reg_size) +{ + struct pci_dev *pdev = to_pci_dev(cxlds->dev); + int d = cxlds->cxl_dvsec; + u16 cap; + int rc; + + if (!cxlds->reg_map.component_map.hdm_decoder.valid) { + *hdm_reg_offset = *hdm_reg_size = 0; + } else { + struct cxl_component_reg_map *map = + &cxlds->reg_map.component_map; + + *hdm_reg_offset = map->hdm_decoder.offset; + *hdm_reg_size = map->hdm_decoder.size; + } + + rc = pci_read_config_word(pdev, + d + CXL_DVSEC_CAP_OFFSET, &cap); + if (rc) + return rc; + + *hdm_count = FIELD_GET(CXL_DVSEC_HDM_COUNT_MASK, cap); + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_get_hdm_info, CXL); + #define CXL_DOE_TABLE_ACCESS_REQ_CODE 0x000000ff #define CXL_DOE_TABLE_ACCESS_REQ_CODE_READ 0 #define CXL_DOE_TABLE_ACCESS_TABLE_TYPE 0x0000ff00 diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h index 4da07727ab9c..8d4458f7e45b 100644 --- a/drivers/cxl/cxlpci.h +++ b/drivers/cxl/cxlpci.h @@ -129,4 +129,7 @@ void read_cdat_data(struct cxl_port *port); void cxl_cor_error_detected(struct pci_dev *pdev); pci_ers_result_t cxl_error_detected(struct pci_dev *pdev, pci_channel_state_t state); + +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u32 *hdm_count, + u64 *hdm_reg_offset, u64 *hdm_reg_size); #endif /* __CXL_PCI_H__ */ diff --git a/include/linux/cxl_accel_mem.h b/include/linux/cxl_accel_mem.h index 5d715eea6e91..db4182fc1936 100644 --- a/include/linux/cxl_accel_mem.h +++ b/include/linux/cxl_accel_mem.h @@ -55,4 +55,6 @@ struct cxl_region *cxl_create_region(struct cxl_root_decoder *cxlrd, int cxl_region_detach(struct cxl_endpoint_decoder *cxled); int cxl_accel_get_region_params(struct cxl_region *region, resource_size_t *start, resource_size_t *end); +int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u32 *hdm_count, + u64 *hdm_reg_offset, u64 *hdm_reg_size); #endif From patchwork Fri Sep 20 22:34:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808719 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5278918C93D; Fri, 20 Sep 2024 22:35:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871716; cv=fail; b=OnmBRu9mwtNhbVmYcTZdldkbYiarggwr4hkjDZCR3ZAy9TWG8S4pqg88PjIK5bTsuXnAOIkU0hajX2Ur+FlksnM9KuEwndbW3sO6pV2LvodUi3uLBl9NjqDY9tXUFSFsVaUYahHvv10X/TfX67tx9RWAgdbBXHZ90l78uG3lwhI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871716; c=relaxed/simple; bh=cOL9T9VqgBapCd3Lkbt5w0Mm1ggmFmBbuHT8D281++0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mVHHcf5shc5fqbac3GP8whaAjNjtHxM8jtsoOOw8g/DKqkmUaXZjnGizmou9yMwrMKzuu5ZOZqBlU6uEnHxEAQcpIBjy+CJqiNDt/v711pLYUdTf8vVKi7ftUMkbTUS5q/y53RsvXvON9y833CUgchrhgLIFsM+PyIuB+Evqtj0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=cphrbCjq; arc=fail smtp.client-ip=40.107.237.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="cphrbCjq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KU1BzTZcexnq19Ad0480PBbP9dSSzvkQ4wHJlSEJL51dU3N9QiV24bN5j/T9sw+Bszi//1LTQCnJ5Rqbn1Ho+iyAdVjGlxnSgvD/DpAiluxmqMd1GGIdcg2d3jaI+XeQx1JG/V9sT3Gnu36HxkvEctHKrCBu8ts02eoBLHOE568UAxTcPJxYvzltnm8F7N2fVFxyZU3Qocuk3sDggPpsQ37ZytQM2W4WH352pIXjWXZSdgVXyC0jRtA9IBnWp4TRWNpfKILnABaBO3GiHzz3ir/dx2JiiUznT+ACHepdtrwi+wqFRHGhKW9ZplCrpVB94FoPspK6v1msxaMQMXUo2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IZjwt1pYXCgMGCEf4dmwKVwcDryN1NIpG6JfXRMHVlU=; b=tBPxW4dHJgtNRSF7YuslsiCMAle7oee+DDsGR1EzkdmTrHlSUj0og12emar+40poFLaKlAjgT4BVfH6FuHQgT702QW/uu8NKtlozYKErZ1UAP74+iO7ziU6BKpP1IDNbhPmNtUKt1ah51JbU78OrLTF4l2b7JFVada3eIMptMmTbkWB8nuSD63LnG01ao6cLjN1PxJQQ251k+8eM1qb10XBf9qMDutuyhtNlJitvMPuyaYs9VhwxoNqwmV+4Nrq+ybJ2BJkdxUyHyzGU8vWxQs4Ns55hH3p7YLjthV3utBOd/28+miGTKZgrXxqkKr/LnmLFPDJVJQAi0llu7YMgag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IZjwt1pYXCgMGCEf4dmwKVwcDryN1NIpG6JfXRMHVlU=; b=cphrbCjqv1mosCC6held0g4xl50yLAb2q38fEiWqSGsoIz4xARkp+D3BufnK03BUvGzQbL3NXVigUnQ20bKU91bqGaCplvxwVGxtjA9ew22grBupYbOs72RTrE9zZ/99WBIrjM2enwyYs4Rd5tOhwzrHRdBlNB36ADyE8mRuDj6tL/XQkdC9FpAseX8IsU7F3oLi7yGzGFzGdfH5PhjMCPrYR5zGr+fLRY5yRczF9xOHm62bFFoMrskT7dTs3uTOyEH8S3yrKSUuS5YtscmoGYxVWQgwZ2ZfPn+N/Kx2TkYmUxvMAmLiVgiToteb9X7yBa1sKsRDJP4jrXoCaNQiuA== Received: from DM6PR02CA0157.namprd02.prod.outlook.com (2603:10b6:5:332::24) by MW3PR12MB4396.namprd12.prod.outlook.com (2603:10b6:303:59::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Fri, 20 Sep 2024 22:35:10 +0000 Received: from CY4PEPF0000E9D2.namprd03.prod.outlook.com (2603:10b6:5:332:cafe::2b) by DM6PR02CA0157.outlook.office365.com (2603:10b6:5:332::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.29 via Frontend Transport; Fri, 20 Sep 2024 22:35:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D2.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:09 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:55 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:55 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:54 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 03/13] cxl: introduce cxl_find_comp_reglock_offset() Date: Fri, 20 Sep 2024 15:34:36 -0700 Message-ID: <20240920223446.1908673-4-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D2:EE_|MW3PR12MB4396:EE_ X-MS-Office365-Filtering-Correlation-Id: 4b7835c5-a439-446d-0c97-08dcd9c47c0f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: hGh7L7DHjy9sBdnJjGWSa5cJANF0/btJDsuXj+kReN7AGjdacc+S0CUPCRxeMOdAjKyg+X5lOFpUwSUPa95mUzenNyg5D7uzV/O04WL+Ywap3hR12SQ99Rkw8oqLyS0rX9obtU5H7Vh0tXU924E1ZT9zXuYoIenZtBCQUhylQCGPrmJ57Pu/YCd1YuAVjnAU8Ck/nCWoPU8XSwpmQ+9L1QVNZ3e3zLbX6mUTDCYcp1+SRhRSI2I46niM2fgU7qpm81zl2Epo/S10qNM1TfWEp7Fv0O4lCkqreq38Wiaso59GePk+j/EB8V8GIuq8+ZuvFy/jZ6+fGEnfdsicV0rCsRgPqluaLqhw5eHX+5O9HwQa/BNT+MjxuM8Ww6jS2Pq3n0T7xj8XZdeCAt+kun5FgM6bC1PkWOakhcCNsG8XWh3gNeeD4U4NxV/TBCkkIAG20CniXQzwg8U/9dXWmB9o0Y7+sOHuHKppuQBj+T+J6Z5jY0jwVVXxEEZRhCVAl6y0eJFv3ChalfsV0LT1vks3KTEd2up99Aj0TAEKSn5O+vRYaUzlZGOqC5vh10xyC++5C1O6CAtBLLVdqmP4VnJD3q8yCWln1YXrJdddJvQ+iDvlg8n3N02+hJ0//39H3bf+ttjLiTU9Kxnz2scjOY6sH7dXyj3SzvmCffixMAI4PqnM777nvdGXJv+sAlKY9sPLOcy7gi996OHguDWD7kUCjytnsS+EfTuFD10QjiJeAjV1nkot10hTRTE1fJj6zKC60di9SkrFtq9rXNZxftIhOjRcFA+aTwQQLOsH5HL7FzdD6hjBY/fQ2ffWh/8+bK+LhUavak+kiMVOJm97f36uZ2lHoSWQOhHy79HoPuym92nfX2fZEr8CyYm8UvqW+z3v8dgvLZrhrwtFg2Fs57+hpFEVLLTl2e9lfhyLp+woKWq4g2gRQhEC+9Z1qQ8cj4M3kJGjnmXp6rI9JcXApA2XCE6We0hlDZRD1VNjQMJjRwIsgDuF3bwkGq5xcQo2B/xEjdYnszQRSDDgq8QN2gS6Ydvl/wXQ6OCS5MC/eXTQPj2kDSFA4iRvfwhxog+AcigxmWRfrPeqBHjDI2jPXRbyr0+G4lKDk0fxbAJCoV70VwWSuvSMG+gNz80tm2bjK2xL7n35R5nYOwhYwnKu5yA5N93tPEgD9KRUd/luQlYT+HTn6XYNyb6cSM3z20vVzC4aKsiPxZv5Vn/zJJkzQr/SWaZcQzYhmjsO0XvZatR4wFSNrbgYncKc4R591vz2n1XBKRnlsIZNRe4IWBKlqC8wF/jagVwYY/n8A7jf3RFiUXB8QZR0iPibAl/s93XC4ab8seDJfxkeWE7Z5vRM9TLssgGabtMjJoemH8Co1m4NQVyR/Gc5JDpNb+jHdrB6d3d8 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:09.7008 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4b7835c5-a439-446d-0c97-08dcd9c47c0f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4396 CXL core has the information of what CXL register groups a device has.When initializing the device, the CXL core probes the register groups and saves the information. The probing sequence is quite complicated. vfio-cxl needs to handle the CXL MMIO BAR specially. E.g. emulate the HDM decoder register inside the component registers. Thus it requires to know the offset of the CXL component register to locate the PCI BAR where the component register sits. Introduce cxl_find_comp_regblock_offset() for vfio-cxl to leverage the register information in the CXL core. Thus, it doesn't need to implement its own probing sequence. Signed-off-by: Zhi Wang --- drivers/cxl/core/regs.c | 22 ++++++++++++++++++++++ drivers/cxl/cxl.h | 1 + include/linux/cxl_accel_mem.h | 1 + 3 files changed, 24 insertions(+) diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c index 9d218ebe180d..7db3c8fcd66f 100644 --- a/drivers/cxl/core/regs.c +++ b/drivers/cxl/core/regs.c @@ -364,6 +364,28 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, } EXPORT_SYMBOL_NS_GPL(cxl_find_regblock, CXL); +/** + * cxl_find_comp_regblock_offset() - Locate the offset of component + * register blocks + * @pdev: The CXL PCI device to enumerate. + * @offset: Enumeration output, clobbered on error + * + * Return: 0 if register block enumerated, negative error code otherwise + */ +int cxl_find_comp_regblock_offset(struct pci_dev *pdev, u64 *offset) +{ + struct cxl_register_map map; + int ret; + + ret = cxl_find_regblock(pdev, CXL_REGLOC_RBI_COMPONENT, &map); + if (ret) + return ret; + + *offset = map.resource; + return 0; +} +EXPORT_SYMBOL_NS_GPL(cxl_find_comp_regblock_offset, CXL); + /** * cxl_count_regblock() - Count instances of a given regblock type. * @pdev: The CXL PCI device to enumerate. diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 5e2b5b3e8f38..33dfdc278b47 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -300,6 +300,7 @@ int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map, int index); int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type, struct cxl_register_map *map); +int cxl_find_comp_regblock_offset(struct pci_dev *pdev, u64 *offset); int cxl_setup_regs(struct cxl_register_map *map, uint8_t caps); struct cxl_dport; resource_size_t cxl_rcd_component_reg_phys(struct device *dev, diff --git a/include/linux/cxl_accel_mem.h b/include/linux/cxl_accel_mem.h index db4182fc1936..6f585aae7eb6 100644 --- a/include/linux/cxl_accel_mem.h +++ b/include/linux/cxl_accel_mem.h @@ -57,4 +57,5 @@ int cxl_accel_get_region_params(struct cxl_region *region, resource_size_t *start, resource_size_t *end); int cxl_get_hdm_info(struct cxl_dev_state *cxlds, u32 *hdm_count, u64 *hdm_reg_offset, u64 *hdm_reg_size); +int cxl_find_comp_regblock_offset(struct pci_dev *pdev, u64 *offset); #endif From patchwork Fri Sep 20 22:34:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808722 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2049.outbound.protection.outlook.com [40.107.93.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DBDC18DF69; Fri, 20 Sep 2024 22:35:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871720; cv=fail; b=tJSie1R74yeZBqKxvLpZHmrjepLYce+cm48D4iNfcqRY58C4Fp4HPZAs3UMYSj/XlFWxExwjrD/L29uLIlSPLCAXe/XZshIlkJUHQnB6FjLPj4z0hMMuocO1vcmHhrrYNahH+/I18k6IimcWz6UjkA1fODYDX3Pqu/X6XDGi/iw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871720; c=relaxed/simple; bh=M0tXuuOTUmKtr2slj/gpiKooYt/SGRosLgIydmlyDj8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=I/gX/NU2utoaxldwvSK3VwI8x52KqJBHv+pIVsdqbclfwd2bIJnDW+6/XT9DRcy/6Dx1PFBnfVPV1XCaCjMN1I0SXO/TC/tpjqnl5UtwjTsLid83qMfwayenqdTQDCZEJQ9IBTWnfkZ/dnvtdPN+8GSfMlqLJMoKOBHhI4/H8Y4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ThDQ4eQz; arc=fail smtp.client-ip=40.107.93.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ThDQ4eQz" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cBhR1zL/Ah9Hnoqcvp4XLne+0esJCDd+NY0axjVHZUbesOAJjin6zUKEhSWQpBR1cp6H3svYjyO1rliankZcRDC20o7PZ853if4srHwi3VBbfW5TKPwjkgxzntZRUsI+SCAx42pugnT50+tb0oTBg4NN2V94ha+lX82OCNBhgEoaj2rdjNLOERnKb6YEsKGNif/doNl2argwexbdlFvAg/WkiVWD8ylZBjjqt1JAegl2LLiVjO6NXAE2DXZpphpOoheAJDu3gvCe/s1PZc9rk72CLZ4DJWtFAloypNYtMX82kwsyW2TXnYiHC9q2vUDu2kpTqL1BYzKi1V1zs6P3BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=63p5Sd9Na6a/+HKoJ0aa+fo4s9+yuIokVvxgLmpdMoE=; b=Se+DpEKpqd54tF+EsXkeg9RamkEhf+i9Kd+gKUs9SyJd6AipvZwz/SsO7as+n2pjkJzIzDn/rw0w5npXTTLU1g4IUhA6Cdf3cI+kfZu84XwW2gIYZ0WdScwliUGQ0c5TXpeIPWijMwbL4xn9wrEym/qRoAynIyoo9saHj33FwccGgT01FrfTg8aBQlMDVsTh/TSmdHFIAGp+efzAmF6F7KKe5ZVkGv7qDKzUY845FFmP8tTJH/w8Xklg3tm+BXNUXxpespPcOVJ51I7EA0BHQ71YVijxX0OeRjuehk7qPjwenKTElXsg3Pm2fa7BDa5tluR6hmVUZv7bLKf/u/9cog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=63p5Sd9Na6a/+HKoJ0aa+fo4s9+yuIokVvxgLmpdMoE=; b=ThDQ4eQzjcR7d5XnRupbEF7y9GMIgvRTvNTlt+f/XCsCQkHymthNxNmU7B5xvoieMt7vDn68jQbjVwZQ52MAqfV5FEleE4bCqZm1RzJJ6K+zhisUepPS5d4pemJP3Lj3K0wu7VKJMM8USsaK/CGRRjlx0gGg/Gb/ZCpSZ1ek/pdcuHIm/yv2XW7TMO4FYSib7my/AEouc6gxqvAyjruIO2lbvWT+LDpZXv0RzO08+yek5arI+W3wZcjqklhmt1a17yl7/IAApsgrgNmn70eDc868v6RL0iIVlMqYDnXsoRt9Q2etW2XibLW/b6T7zqjneVc+2+y788koYyLWT0p9bw== Received: from SJ0PR13CA0192.namprd13.prod.outlook.com (2603:10b6:a03:2c3::17) by SA1PR12MB6701.namprd12.prod.outlook.com (2603:10b6:806:251::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30; Fri, 20 Sep 2024 22:35:11 +0000 Received: from CY4PEPF0000E9D4.namprd03.prod.outlook.com (2603:10b6:a03:2c3:cafe::c0) by SJ0PR13CA0192.outlook.office365.com (2603:10b6:a03:2c3::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Fri, 20 Sep 2024 22:35:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D4.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:10 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:56 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:56 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:55 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 04/13] vfio: introduce vfio-cxl core preludes Date: Fri, 20 Sep 2024 15:34:37 -0700 Message-ID: <20240920223446.1908673-5-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D4:EE_|SA1PR12MB6701:EE_ X-MS-Office365-Filtering-Correlation-Id: 75265c47-9c5b-484d-8164-08dcd9c47c51 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: iFBjKkZd6o5JqNHk5v0VgtrygibBG5kDUOIy8XTUhIXGavpH/e1PBlRg73tsQNZbqELq84b1wr3135+hfOWiYNN5pSH8OrGVKrkw8Omwnon971H1pqoJxQjzfjVuMc1LIUv1t6QkVaBz3P2B/zZQHuWR0ds1LRUxWHB8L1Vh3zS3L4owdayyOVZf5IwMHlGQKZcH0NX2t740OT/rjybIiy5t3SmXOf0OwKlCkpwkTI5lAOXlZ/KlVQmn5ztOq8+hYxnIgKF3vJnYU1q80QI5XdNgg4GfBYnNP/zmR3V+Ulf31FPpzmyoYH6IpZ3jPuyzCUTVrVAh9Z8H0OU4rBABCnbiGlDt+JFu4uduvQ576nVn8YHrIwVVahy3OSNkkcFMQyrPJ2BVFCbCUDlEcIilG7OST/OVSoyKJTGZpvHEF36FZFCUBoCaIz9bWAxfqBtjZXDDLXSJ4PnFZJxYkcGnqutCrzKmnFMrjMDicEQLwfUIMUjxq2ai4CPZzwwWe96dSRn3WbZ7i0rc4Ju5ih67Izm4xU1VctobvR8hFd2F3BMiMFCkF1NZIjkp9eGK3/1x71Wz7JRKweEwsPi68AfvE6MWj+zqLQ0b6fcd86XeYTGvN/LGBR362HiwXt2px8wKnM1OUV1SE7dxTdOI5px8Ds8NiKocA75AVKxneTzWiFX5v9+oFBbVCPuJy9KoGgc0TDdhNJDoVUk8KOmnXPyyEaruXWUymx3PYBnRurflcQ+9bWZFo414UZNRur79oRfsinC8m3ciOwRfh6yjFrjuWsSB2PyXJuqX886/oSpldiIFBO2jRSRHByxriuRJ8nb2IxzB7jEkLNt9jmDguT0CPCiOaDqrLCJMnHbkGxowVoJZXHgfWsTUUBh1081RbumZt1pWZzyCjQPgBfWaXBMuCd7evWyzFMLMc4pR6xiSqtoiFX5HoyabpHWi9zuU9WbXQeFdns/r45G38aAG8OMUerysgX5JPF/MI7WuKqi9zbuURo122LCQpI0wEsp7DX2Ggu+pbuQpkMac40hl46Mu/el88v3UtOV3cdjhjG8JnL+qqsn/0NoCLQ/KE5yh/kaLss9h8+PujCMgxGLVAvvbQ/BzGLaj8ZzZQvGPD/TxtjvGTtLsb8lhDdG7vn3+9TuPiC0XAIN75Umyu6opjqPTlQhybmlGf90ie+LUCI+yyDD7qS+sU81TUy2eLhmQuOMy0SR0SiIYgbXaZZveg6y8NkejO0wQxJ4MRniQf6iUwdlE3gG+oXQ67v+opugZkwpKzDziOmv/NPRvj7YjmhbkXl7VGAiXTDB+cY8xhm1MO5RgX27YmWFr+dPe4g+nmdZb4DjoFHbfGrpdEKv0hqO2wZf073R//pQ9XbYzwmn5BLsBUC9sYrtb1AVUv686r3xtcAQ4dVzrVCu9RKntc5CYm8jpHtysGPfqbwsVRgU9C5q8M5u4WUvk6s6MOIdVVpry X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:10.0895 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 75265c47-9c5b-484d-8164-08dcd9c47c51 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6701 In VFIO, common functions that used by VFIO variant drivers are managed in a set of "core" functions. E.g. the vfio-pci-core provides the common functions used by VFIO variant drviers to support PCI device passhthrough. Although the CXL type-2 device has a PCI-compatible interface for device configuration and programming, they still needs special handlings when initialize the device: - Probing the CXL DVSECs in the configuration. - Probing the CXL register groups implemented by the device. - Configuring the CXL device state required by the kernel CXL core. - Create the CXL region. - Special handlings of the CXL MMIO BAR. Introduce vfio-cxl core predules to hold all the common functions used by VFIO variant drivers to support CXL device passthrough. Signed-off-by: Zhi Wang --- drivers/vfio/pci/Kconfig | 4 + drivers/vfio/pci/Makefile | 3 + drivers/vfio/pci/vfio_cxl_core.c | 264 +++++++++++++++++++++++++++++++ include/linux/vfio_pci_core.h | 37 +++++ 4 files changed, 308 insertions(+) create mode 100644 drivers/vfio/pci/vfio_cxl_core.c diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig index bf50ffa10bde..2196e79b132b 100644 --- a/drivers/vfio/pci/Kconfig +++ b/drivers/vfio/pci/Kconfig @@ -7,6 +7,10 @@ config VFIO_PCI_CORE select VFIO_VIRQFD select IRQ_BYPASS_MANAGER +config VFIO_CXL_CORE + tristate + select VFIO_PCI_CORE + config VFIO_PCI_MMAP def_bool y if !S390 depends on VFIO_PCI_CORE diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index cf00c0a7e55c..b51221b94b0b 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -8,6 +8,9 @@ vfio-pci-y := vfio_pci.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o obj-$(CONFIG_VFIO_PCI) += vfio-pci.o +vfio-cxl-core-y := vfio_cxl_core.o +obj-$(CONFIG_VFIO_CXL_CORE) += vfio-cxl-core.o + obj-$(CONFIG_MLX5_VFIO_PCI) += mlx5/ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c new file mode 100644 index 000000000000..6a7859333f67 --- /dev/null +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -0,0 +1,264 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vfio_pci_priv.h" + +#define DRIVER_AUTHOR "Zhi Wang " +#define DRIVER_DESC "core driver for VFIO based CXL devices" + +static int get_hpa_and_request_dpa(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + u64 max; + + cxl->cxlrd = cxl_get_hpa_freespace(cxl->endpoint, 1, + CXL_DECODER_F_RAM | + CXL_DECODER_F_TYPE2, + &max); + if (IS_ERR(cxl->cxlrd)) { + pci_err(pdev, "Fail to get HPA space.\n"); + return PTR_ERR(cxl->cxlrd); + } + + if (max < cxl->region.size) { + pci_err(pdev, "No enough free HPA space %llu < %llu\n", + max, cxl->region.size); + return -ENOSPC; + } + + cxl->cxled = cxl_request_dpa(cxl->endpoint, true, cxl->region.size, + cxl->region.size); + if (IS_ERR(cxl->cxled)) { + pci_err(pdev, "Fail to request DPA\n"); + return PTR_ERR(cxl->cxled); + } + + return 0; +} + +static int create_cxl_region(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + resource_size_t start, end; + int ret; + + ret = cxl_accel_request_resource(cxl->cxlds, true); + if (ret) { + pci_err(pdev, "Fail to request CXL resource\n"); + return ret; + } + + if (!cxl_await_media_ready(cxl->cxlds)) { + cxl_accel_set_media_ready(cxl->cxlds); + } else { + pci_err(pdev, "CXL media is not active\n"); + return ret; + } + + cxl->cxlmd = devm_cxl_add_memdev(&pdev->dev, cxl->cxlds); + if (IS_ERR(cxl->cxlmd)) { + pci_err(pdev, "Fail to create CXL memdev\n"); + return PTR_ERR(cxl->cxlmd); + } + + cxl->endpoint = cxl_acquire_endpoint(cxl->cxlmd); + if (IS_ERR(cxl->endpoint)) { + pci_err(pdev, "Fail to acquire CXL endpoint\n"); + return PTR_ERR(cxl->endpoint); + } + + ret = get_hpa_and_request_dpa(core_dev); + if (ret) + goto out; + + cxl->region.region = cxl_create_region(cxl->cxlrd, &cxl->cxled, 1); + if (IS_ERR(cxl->region.region)) { + ret = PTR_ERR(cxl->region.region); + pci_err(pdev, "Fail to create CXL region\n"); + cxl_dpa_free(cxl->cxled); + goto out; + } + + cxl_accel_get_region_params(cxl->region.region, &start, &end); + + cxl->region.addr = start; +out: + cxl_release_endpoint(cxl->cxlmd, cxl->endpoint); + return ret; +} + +/* Standard CXL-type 2 driver initialization sequence */ +static int enable_cxl(struct vfio_pci_core_device *core_dev, u16 dvsec) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + u32 count; + u64 offset, size; + int ret; + + cxl->cxlds = cxl_accel_state_create(&pdev->dev, cxl->caps); + if (IS_ERR(cxl->cxlds)) + return PTR_ERR(cxl->cxlds); + + cxl_accel_set_dvsec(cxl->cxlds, dvsec); + cxl_accel_set_serial(cxl->cxlds, pdev->dev.id); + + cxl_accel_set_resource(cxl->cxlds, cxl->dpa_res, CXL_ACCEL_RES_DPA); + cxl_accel_set_resource(cxl->cxlds, cxl->ram_res, CXL_ACCEL_RES_RAM); + + ret = cxl_pci_accel_setup_regs(pdev, cxl->cxlds); + if (ret) { + pci_err(pdev, "Fail to setup CXL accel regs\n"); + return ret; + } + + ret = cxl_get_hdm_info(cxl->cxlds, &count, &offset, &size); + if (ret) + return ret; + + if (!count || !size) { + pci_err(pdev, "Fail to find CXL HDM reg offset\n"); + return -ENODEV; + } + + cxl->hdm_count = count; + cxl->hdm_reg_offset = offset; + cxl->hdm_reg_size = size; + + return create_cxl_region(core_dev); +} + +static void disable_cxl(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + if (cxl->region.region) + cxl_region_detach(cxl->cxled); + + if (cxl->cxled) + cxl_dpa_free(cxl->cxled); +} + +int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + u16 dvsec; + int ret; + + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL, + CXL_DVSEC_PCIE_DEVICE); + if (!dvsec) + return -ENODEV; + + if (!cxl->region.size) + return -EINVAL; + + ret = vfio_pci_core_enable(core_dev); + if (ret) + return ret; + + ret = enable_cxl(core_dev, dvsec); + if (ret) + goto err_enable_cxl_device; + + return 0; + +err_enable_cxl_device: + vfio_pci_core_disable(core_dev); + return ret; +} +EXPORT_SYMBOL(vfio_cxl_core_enable); + +void vfio_cxl_core_finish_enable(struct vfio_pci_core_device *core_dev) +{ + vfio_pci_core_finish_enable(core_dev); +} +EXPORT_SYMBOL(vfio_cxl_core_finish_enable); + +void vfio_cxl_core_close_device(struct vfio_device *vdev) +{ + struct vfio_pci_core_device *core_dev = + container_of(vdev, struct vfio_pci_core_device, vdev); + + disable_cxl(core_dev); + vfio_pci_core_close_device(vdev); +} +EXPORT_SYMBOL(vfio_cxl_core_close_device); + +/* + * Configure the resource required by the kernel CXL core: + * device DPA and device RAM size + */ +void vfio_cxl_core_set_resource(struct vfio_pci_core_device *core_dev, + struct resource res, + enum accel_resource type) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + switch (type) { + case CXL_ACCEL_RES_DPA: + cxl->dpa_size = res.end - res.start + 1; + cxl->dpa_res = res; + break; + + case CXL_ACCEL_RES_RAM: + cxl->ram_res = res; + break; + + default: + WARN(1, "invalid resource type: %d\n", type); + break; + } +} +EXPORT_SYMBOL(vfio_cxl_core_set_resource); + +/* Configure the expected CXL region size to be created */ +void vfio_cxl_core_set_region_size(struct vfio_pci_core_device *core_dev, + u64 size) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + if (WARN_ON(size > cxl->dpa_size)) + return; + + if (WARN_ON(cxl->region.region)) + return; + + cxl->region.size = size; +} +EXPORT_SYMBOL(vfio_cxl_core_set_region_size); + +/* Configure the driver cap required by the kernel CXL core */ +void vfio_cxl_core_set_driver_hdm_cap(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + cxl->caps |= CXL_ACCEL_DRIVER_CAP_HDM; +} +EXPORT_SYMBOL(vfio_cxl_core_set_driver_hdm_cap); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR(DRIVER_AUTHOR); +MODULE_DESCRIPTION(DRIVER_DESC); +MODULE_IMPORT_NS(CXL); diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index fbb472dd99b3..7762d4a3e825 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -15,6 +15,8 @@ #include #include #include +#include +#include #ifndef VFIO_PCI_CORE_H #define VFIO_PCI_CORE_H @@ -49,6 +51,31 @@ struct vfio_pci_region { u32 flags; }; +struct vfio_cxl_region { + u64 size; + u64 addr; + struct cxl_region *region; +}; + +struct vfio_cxl { + u8 caps; + u64 dpa_size; + + u32 hdm_count; + u64 hdm_reg_offset; + u64 hdm_reg_size; + + struct cxl_dev_state *cxlds; + struct cxl_memdev *cxlmd; + struct cxl_root_decoder *cxlrd; + struct cxl_port *endpoint; + struct cxl_endpoint_decoder *cxled; + struct resource dpa_res; + struct resource ram_res; + + struct vfio_cxl_region region; +}; + struct vfio_pci_core_device { struct vfio_device vdev; struct pci_dev *pdev; @@ -94,6 +121,7 @@ struct vfio_pci_core_device { struct vfio_pci_core_device *sriov_pf_core_dev; struct notifier_block nb; struct rw_semaphore memory_lock; + struct vfio_cxl cxl; }; /* Will be exported for vfio pci drivers usage */ @@ -159,4 +187,13 @@ VFIO_IOREAD_DECLARATION(32) VFIO_IOREAD_DECLARATION(64) #endif +int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev); +void vfio_cxl_core_finish_enable(struct vfio_pci_core_device *core_dev); +void vfio_cxl_core_close_device(struct vfio_device *vdev); +void vfio_cxl_core_set_resource(struct vfio_pci_core_device *core_dev, + struct resource res, + enum accel_resource type); +void vfio_cxl_core_set_region_size(struct vfio_pci_core_device *core_dev, + u64 size); +void vfio_cxl_core_set_driver_hdm_cap(struct vfio_pci_core_device *core_dev); #endif /* VFIO_PCI_CORE_H */ From patchwork Fri Sep 20 22:34:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808720 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2078.outbound.protection.outlook.com [40.107.237.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9445918CBFC; Fri, 20 Sep 2024 22:35:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.78 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871716; cv=fail; b=kvAGBqjUcY5Yrx+KHQTog0+IultW9y+lZlLsx9D/4EIm1ycf92G8Sa8dVFUp/KdHax+3A4k69DFW9vy3L+7lo7mR2CPupePaP5IIc6AyAkquieCjdrbdoxLgqpv5WgaA8oK327yxwyMM7RcuC5/FAzQeM9t9eTpEIQtbufNpY84= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871716; c=relaxed/simple; bh=ZiXGEJG6xkMdgDE9njcmLcTrBcbZd8lwOfq/aTyi5xg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=O7EmAYLx60LUzaj8yl11ALaEGSk3fCkW0zrpWohmhJmuix3Ql7G3YpKYlI+HPnfiK23UyteVNG5Z0DVkKj4yUl1QHrFgLaxFoFLEMZaOhtWlPFtddUkiKy+1/I2/9A9rX8f8hd5hvw6k9ZAv1PWlRyhbrYGaAJa5U66k1S0PRmM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=M9XmGkpN; arc=fail smtp.client-ip=40.107.237.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="M9XmGkpN" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fjV9ghtRgVPArNTpjtkP08kQCYlkawhz4MW1P6UKZICI06YcnF1jOC6ZQWqTjVrtddQ7SzrX4jag7PadGamWhYP45lUjgVAFtcOyUFg3W9SsOeBpyNlw6UqqYl+2VuxVjIR9jE3R+GAtppGXhkroGTBvPk+txv7h/ASZmb4PJ0/bHIjJc1rL9qQ4vb0mOi+A+b4ZGJmtjJZVNh9BSjSa2zEzE22E0UYTpaoWvm4rBiWUhOSN0Lk62jjwThaJusO9P+lLqTwpxkWV8Hdr05/Ukvz+sePyRBFZlm3dO7xZ4Wt7erzlyOSR0Acydm2E9X5a7rbyOYukeG8aZF5OW48w1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fdlEmKOktwQDHJltnm7YMmchTX+TSdcBoUqkAglqwl0=; b=cQ+cZCFSvW4IOaouUtWa5KH7jtv1l7O37y4/RniuFH1EyC0x0CM+P8uvt5WbsxlMhibb3cojRjOwaQsJKN3RQ/DKKcGsAD84HxzRDWsBQPll0NBmCYjkUaNvqXf7HvAkezghS+hsFEkYGqGC9CjimFAXA6JZN0h/2QWdz1MjQx187oUtYagm250dhVkiYiMMTFSheHILhaQN5CIv5lTjtHF2fRWdjr53nbYcnbOuzpUQ/z5pqOxHS7iKTTMyX48upHAlozHvKKRCOsgYDcU/nOwSCeaOPDM/DLT6cKCO+vuUUNKJNKFYE3iIfH/9iEcygMevVVnX/vjK7QHALFwqHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fdlEmKOktwQDHJltnm7YMmchTX+TSdcBoUqkAglqwl0=; b=M9XmGkpNTCYrI/ejZA+W5Aq731GPdYUcw6WJhmri+gynwMSODRCDGzaSkdm9mLSENhTQEspo00bNlBIt6nOfDdcaHxeNmyR/OoNXWE0w5f1AAWLyIsIMB/9PryXJpb9yp/z0E3AfNwMNiAW9rR5OGVCRZZ/L3sRjBTf/2vZbXrSpxmuN/kKIcKFqe+0hHZAbtZsqJBfe9UXvHFaL2B3OR0/cLeL27I4GBQL8x8emzAdMJm/BiVAaG4k6Af3StB+kM4/U9YCVIYtm/BkJ5p7PoudL56fNo+xhDlAy0k1TXQuggd6xdiV+9+DMQejCrUCz7ydyKJkS126hdfw+sK0c3A== Received: from MN2PR18CA0014.namprd18.prod.outlook.com (2603:10b6:208:23c::19) by DS0PR12MB8415.namprd12.prod.outlook.com (2603:10b6:8:fc::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.21; Fri, 20 Sep 2024 22:35:11 +0000 Received: from BL02EPF0002992B.namprd02.prod.outlook.com (2603:10b6:208:23c:cafe::dd) by MN2PR18CA0014.outlook.office365.com (2603:10b6:208:23c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.22 via Frontend Transport; Fri, 20 Sep 2024 22:35:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF0002992B.mail.protection.outlook.com (10.167.249.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:10 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:58 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:57 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:56 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 05/13] vfio/cxl: expose CXL region to the usersapce via a new VFIO device region Date: Fri, 20 Sep 2024 15:34:38 -0700 Message-ID: <20240920223446.1908673-6-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0002992B:EE_|DS0PR12MB8415:EE_ X-MS-Office365-Filtering-Correlation-Id: 0d31c9b6-f999-4728-714c-08dcd9c47cc3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: YSuGMpmq1UcZS2H2Zh1/49BIghqnoPNVnArhU8qg9P2Kod0ZsSB73/kPPgVtDsD9UZcg9/CGcmYE3B043n9eveYEMtzJjHcjrSiOE9dV/i7avQ7sXBtgzUal8x3lf9AXP9NEEyy9ZaUcZwpKusl5T/bDxhS3dkwoqGSqnPAcpUbymj+reZbl/T2uAyPd0+ZFfhMpEC1c5QyI5eGyIE5kaubwFSkK9pjLOXIAM1O9jtbJyTZRrabp6nr3j0Zi1R9+KUdInzwu/9amsvQIw+R9c6FBOQKvfDAGflGUzQjUSniXxprUrsmgDPYvSRaasuqvZSy2p7sQZy/aqbFwJ15gjlUI/xZRyR8P12WKvFaER/K5plZ3VPYHQGfFjnpBNfw5yvaOnLTtl8GgUAc3khaggACWpLglKGNl/nNDWsGwEy9a18npZ049RpYOw6XIw7HNN7K7z+0oMYOEtvxLESpofKkdte0UeIg0Bu4OTBbq8dVAuqt1hXLnMrG8s7Pmg03fMeCqDqLc3L0r6wykunfy1dOitz3p9vj36O33NnqeIwI5yVEUTdrW2HQD6tZ90+EeDBv3F/QIanR4ya+LpiE+Gr6RW+5wvPj2lG03Aen8LlFCrxInlGA+gAKP3BzTSXZngJgjuOjg/pqN1BuL7emT9+/+A4uH4WY4KCrFKPRjAUGNSFEf/qTpZIfs8BoGf5EmSir6d14SyVf5O/GdGsivIV07l9Xr9/tuUcSmSSJiXqv0EJ3wVm2uY0ENBLwKutCp5cS+7WLnqlOUICYCQE+zLu57vU/7ERa6fLbrpkN1NzZctCV4BAB77JOa+r2hNG53/zX3dk/J5nLE64PYhrVfJRmFr9uIZIK/T71C28gu6U8SIr1MNoVZRb+wy28It2UhjuH9EGM6IyCHvNTi2LBnvhCbSFo5BYl4HEGBHp1bH+mL19lmx5WH+yLbrFIQqV1Qw4hgRXi7eRrds7A/lGjnmFUtOapmPZMFcWldYmsItnHDqLaLiO9UvHD6DH448cC3iWxA7U8ITpcCC5r8ePY+lMkwvJexYA1+6TlZZ97nfRxfS0HLKBGw340EYRZ2R7UesZ+jRqbQSwk0os5Mu2BpGjUwofmjlFnCOQetSWCu1NBvdO3zOkC3wol2TDFCIoV29H5NHOMw1CEAebF7kDJKDB+vEHTAsihMSgxq7nIN+FwnfPeaMKDma4DD89JNDaXOVaITbRPBvEp+fi9HhXG9xoued9furysvxYW8PoTI8tDKQJPovu4UrSYodvriW40k4J6ozeemKiHGmS8q0NJDkHpJuI7T6ktDOktBtG+e0fqNLwu6pBDKDIaCLyhYPopZ7YM4k+7LM2RbRcsdjg9qj+FRn7ntyQs3r6/1tXj24qlStsPsiCNu8TMdCzqcpyVHwx2Zgd1TiFNwZVbInXrps9t5J7UyvqRmhE8PQw1q/KF3GbolGhIiom8dHwb9hnvE X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:10.7737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0d31c9b6-f999-4728-714c-08dcd9c47cc3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8415 To directly access the device memory, a CXL region is required. Creating a CXL region requires to configure HDM decoders on the path to map the access of HPA level by level and evetually hit the DPA in the CXL topology. For the usersapce, e.g. QEMU, to access the CXL region, the region is required to be exposed via VFIO interfaces. Introduce a new VFIO device region and region ops to expose the created CXL region when initailize the device in the vfio-cxl-core. Introduce a new sub region type for the userspace to identify a CXL region. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_cxl_core.c | 140 ++++++++++++++++++++++++++++- drivers/vfio/pci/vfio_pci_config.c | 1 + include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 3 + 4 files changed, 144 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c index 6a7859333f67..ffc15fd94b22 100644 --- a/drivers/vfio/pci/vfio_cxl_core.c +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -102,6 +102,13 @@ static int create_cxl_region(struct vfio_pci_core_device *core_dev) cxl_accel_get_region_params(cxl->region.region, &start, &end); cxl->region.addr = start; + cxl->region.vaddr = ioremap(start, end - start); + if (!cxl->region.addr) { + pci_err(pdev, "Fail to map CXL region\n"); + cxl_region_detach(cxl->cxled); + cxl_dpa_free(cxl->cxled); + goto out; + } out: cxl_release_endpoint(cxl->cxlmd, cxl->endpoint); return ret; @@ -152,17 +159,135 @@ static void disable_cxl(struct vfio_pci_core_device *core_dev) { struct vfio_cxl *cxl = &core_dev->cxl; - if (cxl->region.region) + if (cxl->region.region) { + iounmap(cxl->region.vaddr); cxl_region_detach(cxl->cxled); + } if (cxl->cxled) cxl_dpa_free(cxl->cxled); } +static unsigned long vma_to_pfn(struct vm_area_struct *vma) +{ + struct vfio_pci_core_device *vdev = vma->vm_private_data; + struct vfio_cxl *cxl = &vdev->cxl; + u64 pgoff; + + pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + + return (cxl->region.addr >> PAGE_SHIFT) + pgoff; +} + +static vm_fault_t vfio_cxl_mmap_fault(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct vfio_pci_core_device *vdev = vma->vm_private_data; + unsigned long pfn, pgoff = vmf->pgoff - vma->vm_pgoff; + unsigned long addr = vma->vm_start; + vm_fault_t ret = VM_FAULT_SIGBUS; + + pfn = vma_to_pfn(vma); + + down_read(&vdev->memory_lock); + + if (vdev->pm_runtime_engaged || !__vfio_pci_memory_enabled(vdev)) + goto out_unlock; + + ret = vmf_insert_pfn(vma, vmf->address, pfn + pgoff); + if (ret & VM_FAULT_ERROR) + goto out_unlock; + + for (; addr < vma->vm_end; addr += PAGE_SIZE, pfn++) { + if (addr == vmf->address) + continue; + + if (vmf_insert_pfn(vma, addr, pfn) & VM_FAULT_ERROR) + break; + } + +out_unlock: + up_read(&vdev->memory_lock); + + return ret; +} + +static const struct vm_operations_struct vfio_cxl_mmap_ops = { + .fault = vfio_cxl_mmap_fault, +}; + +static int vfio_cxl_region_mmap(struct vfio_pci_core_device *core_dev, + struct vfio_pci_region *region, + struct vm_area_struct *vma) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + u64 phys_len, req_len, pgoff, req_start; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_MMAP)) + return -EINVAL; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_READ) && + (vma->vm_flags & VM_READ)) + return -EPERM; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_WRITE) && + (vma->vm_flags & VM_WRITE)) + return -EPERM; + + phys_len = cxl->region.size; + req_len = vma->vm_end - vma->vm_start; + pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + req_start = pgoff << PAGE_SHIFT; + + if (req_start + req_len > phys_len) + return -EINVAL; + + vma->vm_private_data = core_dev; + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); + + vm_flags_set(vma, VM_ALLOW_ANY_UNCACHED | VM_IO | VM_PFNMAP | + VM_DONTEXPAND | VM_DONTDUMP); + vma->vm_ops = &vfio_cxl_mmap_ops; + + return 0; +} + +static ssize_t vfio_cxl_region_rw(struct vfio_pci_core_device *core_dev, + char __user *buf, size_t count, loff_t *ppos, + bool iswrite) +{ + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS; + struct vfio_cxl_region *cxl_region = core_dev->region[i].data; + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + + if (!count) + return 0; + + return vfio_pci_core_do_io_rw(core_dev, false, + cxl_region->vaddr, + (char __user *)buf, pos, count, + 0, 0, iswrite); +} + +static void vfio_cxl_region_release(struct vfio_pci_core_device *vdev, + struct vfio_pci_region *region) +{ +} + +static const struct vfio_pci_regops vfio_cxl_regops = { + .rw = vfio_cxl_region_rw, + .mmap = vfio_cxl_region_mmap, + .release = vfio_cxl_region_release, +}; + int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) { struct vfio_cxl *cxl = &core_dev->cxl; struct pci_dev *pdev = core_dev->pdev; + u32 flags; u16 dvsec; int ret; @@ -182,8 +307,21 @@ int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) if (ret) goto err_enable_cxl_device; + flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE | + VFIO_REGION_INFO_FLAG_MMAP; + + ret = vfio_pci_core_register_dev_region(core_dev, + PCI_VENDOR_ID_CXL | VFIO_REGION_TYPE_PCI_VENDOR_TYPE, + VFIO_REGION_SUBTYPE_CXL, &vfio_cxl_regops, + cxl->region.size, flags, &cxl->region); + if (ret) + goto err_register_cxl_region; + return 0; +err_register_cxl_region: + disable_cxl(core_dev); err_enable_cxl_device: vfio_pci_core_disable(core_dev); return ret; diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index 97422aafaa7b..98f3ac2d305c 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -412,6 +412,7 @@ bool __vfio_pci_memory_enabled(struct vfio_pci_core_device *vdev) return pdev->current_state < PCI_D3hot && (pdev->no_command_memory || (cmd & PCI_COMMAND_MEMORY)); } +EXPORT_SYMBOL(__vfio_pci_memory_enabled); /* * Restore the *real* BARs after we detect a FLR or backdoor reset. diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 7762d4a3e825..6523d9d1bffe 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -54,6 +54,7 @@ struct vfio_pci_region { struct vfio_cxl_region { u64 size; u64 addr; + void *vaddr; struct cxl_region *region; }; diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 2b68e6cdf190..71f766c29060 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -372,6 +372,9 @@ struct vfio_region_info_cap_type { /* sub-types for VFIO_REGION_TYPE_GFX */ #define VFIO_REGION_SUBTYPE_GFX_EDID (1) +/* sub-types for VFIO CXL region */ +#define VFIO_REGION_SUBTYPE_CXL (1) + /** * struct vfio_region_gfx_edid - EDID region layout. * From patchwork Fri Sep 20 22:34:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808721 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2069.outbound.protection.outlook.com [40.107.220.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C57F18DF66; Fri, 20 Sep 2024 22:35:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871719; cv=fail; b=o7tryRIi6HF/j9vylWr2kAkho0f9Jzuya5H2x1uHOK0C8ccUDUU/1Z55wEpeYCq3nBMAbUhp2Pt0Ox5qhjrz8UqfaNkF4n8DLP2MFJrhdHwPzO/BFae/XtcfKyIQXeyBo1wT02vyjDqfN3PgNOklLs0r//ycvV3obY1GopmSjfA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871719; c=relaxed/simple; bh=3s6iDmvat1DTIh52UuYQFP6Eh5RHgyLznQTbt3wPVfQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=TO0uXNrmWGOQbK5kGPadFaT9IY1rtvEOLU7GS5dt+8UUfNYR4FnlYgaoOQuTwUOCS1SjUn1Af0E/gwaPVXbLzZun+HDAE5vVo4AZ6CeM+Nnh2IoXNmzzOQIE5Z/12oCbM5NTYG0Sz39eirs3n+2gaMttbX6wM9j/+iqapeqp+oQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=OV867Uxy; arc=fail smtp.client-ip=40.107.220.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="OV867Uxy" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=j26JsOtsuPtjvQUxs03O9bF/cEFUaCOd8IEFhUXx2AwYwFDlnQqkM8Eb9fR43M46z0mJlvWzSKVzGaAz3n4pCAtE+3w7fFckALS6MdSlwBZMYEkR4v0usG1TqIu1iSb4vkATpRWZU8mgIR3DG5R4M0yjNyESBWVHn6nrtqb2chsD7w+VHurhikCdVhi+5au1ciVCFk/qFeOASpfM0TilMvmVN+lkvW1YMi9kwq6XTdfBqhhhsKVs3gM01T1+ZXXsIJtrHjnNgNCU1fauV7YnKjXs0oGalVZBlb5a/LTSUXx9v616zggR2ApS2f/31tXYjIpYVyV0XTQXJnxWb0vScw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JV+Y4GHVgBeatSHjmQi3GCXcGD0XiYZoTobGPWMYW0w=; b=F0+/LR2XqoHRzvjh1dZdg0Lh6oSv4ToOc1o/2Lg7f7tIXIkMtf0NIUrWBA+8BEIPXNpwXJs+Mkj+mMjO9+MFk7h3GJ7C6DM65eXw5dsR1oeILc0d0OZRoOiqSUzSO2fJ02QS7KFpFM6lx5z0dZKTmGURvWvqygoM8eDeJ+za5Fhpd2LN1hONhMv+kReRmC8kt3nFB5uuSd807mKJwpWnxBP357Xv7WvXTUrmQDKJM/02V7Tmnjp7gJKEFT1ymeb6xInYy4PfniqXgx/KkDWsSD/XKtPDpx2FRscUrd1d8g6CXPCBhcsN3QtqS2MROKyINsnL5IN+Uw2RiltIAZ97fw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JV+Y4GHVgBeatSHjmQi3GCXcGD0XiYZoTobGPWMYW0w=; b=OV867UxyjSs0LBKr/or30aqlRXtCGayaoc9si0+buFYA1iKHpjPtguIxuPxGHMCN3aVcRVS/HL0IYB/BQp5os4HLNOywkQOHlqM4A3JjlNXp1gYWkmnIYDff07r0DcgxLxz5Z6oarle/TtJ0Jjt/lfLPxJ9XstgOdTZU6TzMzhiqDJZQUXLAsMZNgTnKoPkMJGw3Hk+6XGmV2GPiERlRCBBw6/xssJ4L8rjKP9rO1BpuyObwDlQYzu+pnbIP2U7kVU4J5wsw8Q3ccw+MVUZGFzc3/2ymESdHm0Y9E9HGKBlROTEDHf9vWkKyZRD9vHJMqNqO8cza4yD/4FDZ8nrlRg== Received: from MN2PR18CA0019.namprd18.prod.outlook.com (2603:10b6:208:23c::24) by DS0PR12MB7654.namprd12.prod.outlook.com (2603:10b6:8:11d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Fri, 20 Sep 2024 22:35:12 +0000 Received: from BL02EPF0002992B.namprd02.prod.outlook.com (2603:10b6:208:23c:cafe::6) by MN2PR18CA0019.outlook.office365.com (2603:10b6:208:23c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Fri, 20 Sep 2024 22:35:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF0002992B.mail.protection.outlook.com (10.167.249.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:11 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:59 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:34:58 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:57 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 06/13] vfio/pci: expose vfio_pci_rw() Date: Fri, 20 Sep 2024 15:34:39 -0700 Message-ID: <20240920223446.1908673-7-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0002992B:EE_|DS0PR12MB7654:EE_ X-MS-Office365-Filtering-Correlation-Id: de1e1571-85bf-4dd7-a5d9-08dcd9c47d59 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: OkkQAh4gOGSmRqhFTHXY/j/D5eZH8D2X0HKtfRj4LTUqomP7qNbVIl4BMs13pKGhhLfNxMCWpwQVCsa3czf1il5YYJ06ev63fF9/YfumeJJlc/HYYmNNnVFKv7aapT86GgayxqpO0mzjY3yKYFGh89VK+h8Aeb3VD8tfPFrcH+lzj+Y+HKfOBGgM4o9GOJPG1uQVP4hhI5TusATG1BRlKoHs2OC+DymN+7EUHsIKHisUwhb8ynVj/Rnr4uRjuvhrKihc8tE3yHQHrXqvRldqODkc5xIeTXv9FBEaOs0LGQbr2PBd4l7dcv/FdCal8PvReupa3mAjG2DFp7D+k/4jci9+gCz0+kuh4aMcKf3p+ZKiol876A2f0UVnJF+DWTt6k/5MbCE3Sbx9qWBC8v+Ae5b3/gpQ9dv5raME2RrgpgBAGgAS9jk3YeQ2J2VHqJ6GcojQPdv7m7IkEaIfyUIBl5xfLDg5fHqs77KV7WR/gleGzMqCObyTrjpjTuXThLa0P4bl/i3uR5ZjE0ecNrh4/1G7EYyXVOom96q2c7E+AV9kBejkLi8Eokgt3M/BdNKbqikjw5YDZmtS8zEU2qgUgxrOzvlr9aIPZtpcHqyI2/ycZyZoL54s9pfnXw13eW4itYdy/Dbz4DxhK7yA2l+HMKq0gOCAmoUqyBmeUnL/NpRJl+uEfdqL6yGGgO4gyyFBjIEZAmimG2qHAkmX8R+1RYKL8RHu8QNRtSRU5sVlv5zLlTyfehc7ZiaSx8NFQFuQer79UUiL26GRgO/QYrteRUpMGowmeJ/+rAGC8Hu+oVPAWZL8EiykWLWnt/hulstaqcw5BHnnP5na8gOvh9uHwX3oqIymouV6nKxdjUE+YeA/fLGPGPe8ulRsPviDRZRon6DOz1SdgO+qUx3nXSUH3ZXeQcH/IPscmQ7i0ySDuhWD6VcVIlvaH9VFwKQJ4fIdSKy6RO669FXiiRtY4T5hY2AEuSuWsAFkPx4wSmjU+vDymYNazWDOSUfZ2eOKAWdRi0FZXUKbJ+nbY4n999c0r+l6C/Y3/OHi1+1zOB+TrHIZ40On8XtTe4UWxipNFqlTGmYd4/EGS2i0NjG3cB760R8znevmvacIYXcMUDqCRXFJ7hLAl/uTwUKXgJavbibtWVxfnFxvl9NLeNDj/b2vVAHbCqg2bSQdzddGzrzurOi7+X+cL7JH5RsK60fmJADaWd6RPKDXnAry+gW0wGLN/OD81b4+ZA0WSZw8Ksg9DFS3W0iBZQXkxMfjces3nVjHwhLXQfXFBty2lF+iXWQHJojmeda5UNb3MNeFCzinI80KASu05MZ0JNKGxEro6XRAx9WbvmVFJfLbbw4VpErPNQq2K5N7hvalV4WqkcbaKcRqELbUT/9tEClsuQwny1iIeR1pmvVPYVqimnhzvNpyXBHmW2oOQodyePly+N+tYAtBynW2u1OwdHf4QUuM5nq4 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:11.7737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: de1e1571-85bf-4dd7-a5d9-08dcd9c47d59 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7654 vfio_pci_rw() is the common function for handling PCI device read and write. A CXL device programming interface is built on top PCI interfaces. Expose vfio_pci_rw() for vfio-cxl-core to handle the access not interesting for it. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_pci_core.c | 5 +++-- include/linux/vfio_pci_core.h | 2 ++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index ba0ce0075b2f..9373942f1acb 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1537,8 +1537,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags, } EXPORT_SYMBOL_GPL(vfio_pci_core_ioctl_feature); -static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf, - size_t count, loff_t *ppos, bool iswrite) +ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf, + size_t count, loff_t *ppos, bool iswrite) { unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos); int ret; @@ -1583,6 +1583,7 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf, pm_runtime_put(&vdev->pdev->dev); return ret; } +EXPORT_SYMBOL_GPL(vfio_pci_rw); ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos) diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 6523d9d1bffe..62fa0f54a567 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -144,6 +144,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, unsigned long arg); int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags, void __user *arg, size_t argsz); +ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf, + size_t count, loff_t *ppos, bool iswrite); ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos); ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf, From patchwork Fri Sep 20 22:34:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808725 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2046.outbound.protection.outlook.com [40.107.94.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A10ED18DF93; Fri, 20 Sep 2024 22:35:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.46 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871723; cv=fail; b=kPOcVs7fD6ANC8sNdqF0kS4p5SyrPQzavkis7UsAqOr5c8miNvCp370aZmn95+svMCC1YC/Hu3AQ7xiK44GL4EVEMN7/+vHW82AiPgqK5dzy1f9Q8XpGgJPSnQ3Uqb1KH3XjsdGzJobd5GRQTWZxqggnvbKCnVhoaKHAOZquwiY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871723; c=relaxed/simple; bh=9iQqp0Xg16+bvOvxGOUK5RL3LGwnUWpBuuXcFaZe7as=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BfaKSk10UOcpHndtW7lS8jYbyDxHdXbJpNbKtiY7yunI3DdFWHQoQNm1SmtE0NbmIuNTVHMnt1h74J9mGSVfR2ogMDI1vFy2omIkZoVMJd6k/0y1Y7yms1mdJCHzEJVZApf91tYdgu1nYbNDnsHAbGZ2u85cEClmHlhGtZm/XVw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=M5nEo36D; arc=fail smtp.client-ip=40.107.94.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="M5nEo36D" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tY1Y6ZQaXQCMWIg3lnHaueLP2NpR0eU7qZPqEFKEzqIdzUAjWiM2hJbdYPYbe/Lhplq+kS8m1P7s4o4+FjaUkOx1zpYdQQWdD1grw2JXUjDH6fcga6myjElrgsYly45GmwyQW5HIW8ncEeMU9OT+cjUMrwhsIq0AuTLtEAUfYwdZtr8bV10/aJ1Bk8MxfZuhj66BFo45Vpi60bfNAiWfeu9eEyuzgS+lRNZEHGAouGxKwTnx+ln5VO9E9+vqBvyQuXoyOUBhEWVYdSWpE19JFFoCwDGD1+1dy1XyQQoUPQL6dnROX4aU3EaNPrRwS1F49zsmJMR1l44PowcAWwLeUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=L659QQc2Yqg/ZLysXrPLTj7fZzo3aetDWC8XIgo2i/c=; b=L/TJhUC30sn5yP5GqQXqy8sn2Mbfw+CQoUhFyt9acVgXXNEMGKBwnx/TwlRep2p/oKg+FevaVL4ACLIfK+aJkiYp3siosxFcR4zAc/AiS6MvtHp6paq5nr5nHbxd5y0SnghB66/445n16Ps2phCjeoAkAOluWBA/RE2HNmFpcHxgiU4v/tivrfz3j8hpTAK7mf65A/MFLmDTGW1iDaCoALNJJhrf3zi8e8R4ugVV7z+qzDkeK7ebKCiNz8oWUSSjSirqerI4ZW+bLxGPNxjhZSPUQK7Dbyz2DYy2RDr4bfT5Hf7pne6kEyTHo5Dp+ajs+uMnG7exUGt+s2GaB1UmmQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=L659QQc2Yqg/ZLysXrPLTj7fZzo3aetDWC8XIgo2i/c=; b=M5nEo36DmK7I2z7cm/9szHo2/yxK0OGRBvxkCxZe8Bs0Z5TXUvJuustNpevn+NtI9K9cqFaGZdLgpbq3OCLvq4QMvdbEUUzsUZXyIYfDpomhUJmDli9q+1W17PzIP+fuN5zs60kVyZQMa/e1xwFGq60gMSx9X8+ad3X/tEXhHeyUU7Dqz85md08fC+A9xnfExDmPPjrjKq84xWFLWDi/en17VX3OsOWbhCueeNLWNWYOXzGHUs3SUHk9n0pBC/TDKjGAjpX4wMFb9UsSdkHpXnDde1IQsXIRPKaP2ULiUGNMdO05a0ePrWOOhli1rigNv0qAM7/TT73blfPi47DRow== Received: from MN2PR18CA0024.namprd18.prod.outlook.com (2603:10b6:208:23c::29) by PH7PR12MB5903.namprd12.prod.outlook.com (2603:10b6:510:1d7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.17; Fri, 20 Sep 2024 22:35:15 +0000 Received: from BL02EPF00029928.namprd02.prod.outlook.com (2603:10b6:208:23c:cafe::a8) by MN2PR18CA0024.outlook.office365.com (2603:10b6:208:23c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.23 via Frontend Transport; Fri, 20 Sep 2024 22:35:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF00029928.mail.protection.outlook.com (10.167.249.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:14 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:00 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:00 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:34:59 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 07/13] vfio/cxl: introduce vfio_cxl_core_{read, write}() Date: Fri, 20 Sep 2024 15:34:40 -0700 Message-ID: <20240920223446.1908673-8-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00029928:EE_|PH7PR12MB5903:EE_ X-MS-Office365-Filtering-Correlation-Id: e77516d8-f883-40b8-a900-08dcd9c47f2a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024|7416014; X-Microsoft-Antispam-Message-Info: yP/8Bjf9ZzPlcZDrax67rK8qe/WhXEz2m2MHpII7D0n/xzZmjVJiu7y/ViOo2pgBeahLKxyEZzXsxLAlO13tUMn78dLi1cR4Ba54jNlapW+GHMbe+reKmuyos3zrVT013XoDKPw/366V4+JuRXyClej/hjUmJX5+sMCfbdYEM95uhvYGiJjM+ufW0h/uP309xgn9JTOyIKuMPzlGheJL0Byn+WEnj6LOUSmYk9scFNYfztT12dm5G+zFScl8jlF+ZuVTUakHL3qTEY/EA+PlK2LABb5/WGPwxq+NFsEMgxGeTDGQlYKywFs0Jhet+uFwmRFVSRslBnIBD6rQhokrNzvqsKRG5gYxM7MxlclVZzXJ4Wb3oolFj38HsH0jLj2NFWpZFL3VKKTnuRyZXD5svvV1KabA9GwSnsaowE8S5Fqp+NhVbze0bQpKuu6uKYLuVXDAcc7FbcvVBJrtL+wT6EdmO1FKZYFDBquBsqxLFqMYdswnYiYaPIijhRNQFeHyoqfRzn2c3KXxzhGzLuknCk3/HVfzVvqJUc6af84BEyPaBtR1PCc1/KAwYaJ0J00xlOHKPBO8ZzNChrz5Yq61pGu8GRUiYcllodc6hbQQHBwnpMyn7YJJlNaKUtuekIejb+0eWMWKrjaW7XcVgNPTh8/0XbWA1pZSOlzKQArrxlNcsZ/BIUn+B6zoAuTnHgTqi6b+TupKfFX5lEPzwDEfjqhPDXU3X+mZe6WCC8HhxwBIlBMa9OqVQn8/isK4pneCFOHpnR6c5bjarmoCs2Ccl8l1MAZRmETXfsmS9FVhwkLqytD/VPTjUtRmqIDGVd8EUPpXMTBG373oQyaIkTLmXLrvO3kWVOF/SrcTRUCIXAl6MJjOPvQoGaK/LlPPaVlbQqgSsjBhZQzaGBOiRA8mt+gn6dQld3Mr/lVsBfteMufUAXP4KYPntd7Avs5mLhfP2hn7eodAc4TWkgjKvYD4he13V2GkR0dwZPRdbGJMHzLHQFvMnGZjfFGK6Fx+gCd59ZXE25S21ssBmu7l/lHRO3uMPI4G7Wp+VZbkkerGHaXE5ogVLwbT6+Ol625VN/EZp66EUQBMTvekKU1+yy7kU4g4HmJTBjVvy4kGiM1ojuirtWPXgD3n1GARAVHYNIbzEGSl9Id69MJmH/3gFAzg5rUBwdBmfe5HZ6Tkk1xMoh73fLxz6qlmmAo4p+UHwkmMWmTtqe/eQHbO06yuKa4N/uR34sUvG7Bv4wLb8vxP+wo89wLy7ckci58DHrmvWXUKlhjIBBb1emgDcOA3Pi7KSnZDmOnL0rnQLeJuRpKm5f7RFwJRjuOYj6v6sNp8xgNvh3zoXVkHZTORpfzINq4/60MXJU6oxcy2GdpAovFc8+03+mjizaOOXre/DjgffV1pltfTqXEYwWj32CxWzm4wE5+Cnyo1gCgTXSn7e4Nr/JeDcq0ABGdI290T2fWoEchN X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:14.8056 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e77516d8-f883-40b8-a900-08dcd9c47f2a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00029928.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5903 The read/write callbacks in vfio_device_ops is for accessing the device when mmap is not support. It is also used for VFIO variant driver to emulate the device registers. CXL spec illusrates the standard programming interface, part of them are MMIO registers sit in a PCI BAR. Some of them are emulated when passing the CXL type-2 device to the VM. E.g. HDM decoder registers are emulated. Introduce vfio_cxl_core_{read, write}() in the vfio-cxl-core to prepare for emulating the CXL MMIO registers in the PCI BAR. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_cxl_core.c | 20 ++++++++++++++++++++ include/linux/vfio_pci_core.h | 5 +++++ 2 files changed, 25 insertions(+) diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c index ffc15fd94b22..68a935515256 100644 --- a/drivers/vfio/pci/vfio_cxl_core.c +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -396,6 +396,26 @@ void vfio_cxl_core_set_driver_hdm_cap(struct vfio_pci_core_device *core_dev) } EXPORT_SYMBOL(vfio_cxl_core_set_driver_hdm_cap); +ssize_t vfio_cxl_core_read(struct vfio_device *core_vdev, char __user *buf, + size_t count, loff_t *ppos) +{ + struct vfio_pci_core_device *vdev = + container_of(core_vdev, struct vfio_pci_core_device, vdev); + + return vfio_pci_rw(vdev, buf, count, ppos, false); +} +EXPORT_SYMBOL_GPL(vfio_cxl_core_read); + +ssize_t vfio_cxl_core_write(struct vfio_device *core_vdev, const char __user *buf, + size_t count, loff_t *ppos) +{ + struct vfio_pci_core_device *vdev = + container_of(core_vdev, struct vfio_pci_core_device, vdev); + + return vfio_pci_rw(vdev, (char __user *)buf, count, ppos, true); +} +EXPORT_SYMBOL_GPL(vfio_cxl_core_write); + MODULE_LICENSE("GPL"); MODULE_AUTHOR(DRIVER_AUTHOR); MODULE_DESCRIPTION(DRIVER_DESC); diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 62fa0f54a567..64ccdcdfa95e 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -199,4 +199,9 @@ void vfio_cxl_core_set_resource(struct vfio_pci_core_device *core_dev, void vfio_cxl_core_set_region_size(struct vfio_pci_core_device *core_dev, u64 size); void vfio_cxl_core_set_driver_hdm_cap(struct vfio_pci_core_device *core_dev); +ssize_t vfio_cxl_core_read(struct vfio_device *core_vdev, char __user *buf, + size_t count, loff_t *ppos); +ssize_t vfio_cxl_core_write(struct vfio_device *core_vdev, const char __user *buf, + size_t count, loff_t *ppos); + #endif /* VFIO_PCI_CORE_H */ From patchwork Fri Sep 20 22:34:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808724 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2071.outbound.protection.outlook.com [40.107.94.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12DD018DF8C; Fri, 20 Sep 2024 22:35:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.71 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871722; cv=fail; b=qfVla5XotKx196L+JsrX1XBXfrFmpPOlHTvD2sfHdxeomCoxqsvG2AbHkyb2qlLK25kpJovOEowGjLXEBh/nkFOL0l20r9f03NQznZtQ4ikiNXmpb9N10rp1r/74KmI/PlXgI749vn/w2w7jdqmeUusvYRbpfyKEVRWJPKZGm/Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871722; c=relaxed/simple; bh=CNypwOQFjOrA9uoujWe1o9nKAHHeK/8cRfipSHQnETY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=krgloT2kxwiBks0CC07LZFWGIi8VlMZ7BRPh6IS91VL26vF8fgSI8lhydp6vVXTQKd9AXGAX6SyxZk1Brq7tZKf63MpJn1nGCazw6tPRmSYvlbcW67avEdBFJ3RLNvcvgAX2OaoNL3gotb/u8pKp9sPJeDM9hzuaolzHymz1Yhs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=UfKFZ25s; arc=fail smtp.client-ip=40.107.94.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="UfKFZ25s" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iD9LSqgL2z8KfWEjbslSmP6ly0zaQSnHdD9vpoQWz54k889Er/ioJYz25w0DGTvgKLa4HQKZoEC5RetbSbfS6c4gxSOGBycVT5Q0OZo/w+N9kP12i8C/iB0dN+vmfuwTQINLSN3VnALCpHaGd7K+MXno1oy5ELHwyARqnIhze21gXEXNwG5NOoFfitNh5memCJrSGpK/rStJkmeBas8fp0sKWKSpa15rISsvYPSfsn9aeSNj0I5loHyOfPpWoduxdODhfRnJ6B9xYGgRmhgg8mIb8TfzataZxnO4WZkY4MhWasS81xXMp3OfDIWMaHtr2IGrfxv6gL90NxK3Lpw8AA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1Lz+MrXoheRZPhqJ3diJ1ne9h8twBQcPiLrAkvWRYd8=; b=lMZ6GIv8pX7scQoHmHx/vPixGgD/JOR1YuxE1wtFDXAdlRGDvEv5q+mvwfSH6e2jg2KIW1GvZFK7CqEigbH2wuhdIpJIezsVUP0U7Bfg7CV8dkxQurjQ1o2fMkqFku8uHOm1RFCh3uQw60bAKdwHzjfnUUjLNV1yAIGB5MMTkBNidKfutMf75fPRtHaEJHCWT84KVR0jmaKidkBlXvEvKz6zj+PXTpyWZ9hb0q1EOg78qMxA5FWNzZA0hNeMJYr9za6rTx7rhp3jzrE/OQyqNyl7mDnG4zcMwpXmhI2HfCe6LWFOVwHsSjTJX1QKi2GXzmONGc19UwxYvju10KVbiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Lz+MrXoheRZPhqJ3diJ1ne9h8twBQcPiLrAkvWRYd8=; b=UfKFZ25sLpPsdmDMTV1HH5PycQZAh+rUXlqEvPzEcAXey8tv7nZT9sOkgWoKtiagg70kjC/w9yu40UNzfiMM7nyYybrYJQkIEu5kvuSrgCPMD3JBVioA25nLo/2qRp153lMFPMFKOAbVaM+FqBNh9aSx45k6bIIEgigN3lY29s2gBzNbggJDOvYaWZAIK65vVr3Vam1M+JwrosYGMUz6GvGiJn4QQ/79xbaeIWwfCa6IgKxiVPCbSK2FtQo/3wYHgix6L2yBc7/R3xWMhPhHX7IZj/Rtaj3EZWhFZimqsavUMXn0RLDOoE/UZxKAoO0e4uxHL6eo+p5l5iF3W7oeGQ== Received: from MN2PR18CA0002.namprd18.prod.outlook.com (2603:10b6:208:23c::7) by MN0PR12MB6367.namprd12.prod.outlook.com (2603:10b6:208:3d3::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.22; Fri, 20 Sep 2024 22:35:16 +0000 Received: from BL02EPF00029928.namprd02.prod.outlook.com (2603:10b6:208:23c:cafe::3c) by MN2PR18CA0002.outlook.office365.com (2603:10b6:208:23c::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.25 via Frontend Transport; Fri, 20 Sep 2024 22:35:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF00029928.mail.protection.outlook.com (10.167.249.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:15 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:02 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:01 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:00 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 08/13] vfio/cxl: emulate HDM decoder registers Date: Fri, 20 Sep 2024 15:34:41 -0700 Message-ID: <20240920223446.1908673-9-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00029928:EE_|MN0PR12MB6367:EE_ X-MS-Office365-Filtering-Correlation-Id: 74d6b09f-7005-4c0c-a1e9-08dcd9c47fc5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: Wb1I7xCBiCBhf3MP1QoeX13Djt+Q1LMlAlCikqMI3bvrWbySKXzKHCrgO5/Tkd5RJ8422ZChbQ8+rgYoFO/ef5+lGkUX+WX3XDGGT/6N22poiT49ZNwlwa6DhfSBIFI8LAFtix6dk8n5DAPS/jZZqVCmkkudeBnf/vJCweOATcNJ+90c2mvqWaUZj9mRPRh6MwTbTGmmVxBZ150ppyi5q4xMPzi1ZzqAr4gnEpJGHz5g1kW2ShrljCDvMhKxt/4vW7TjX/CjqqCTfWYcoCZ7725gCL4g6LRh6jWtoWKgphVxDlADSMadSV2dFYWSUi7vlwoxmArBKlOQXI4gUY7Cm602vEhTcA7s00nz4SkF0PwLIr2I7ldYVYnViOG5XqEmdQ2k8s3RCGOs55v3Ki79+rzj88s2qTZr2Wp4SRywY7StIx/O7p91l8dt5G1mZAZ7LoIpt2XOE3GhJYv2INrhuUSg2pVpA6JPwfKHKFcDwkP9eTnd19zvtomzmJlxQ5rxwmBHn+BN8dcl2aRg0mhNgSDcMYmYFDP6dMETLwh2I2Ah2sCcCVFBAKWPq7bPaS5qVzrMpCCHxc3QUiejpGLTkKdvLd4B61HUjRZoJmPjvHZx7DQQ377ihw+0OTL9/INoAwQFRqOjVMwhuBRwOxG2J+WNwykinpG5X+C1C8HItUigPwT3uZT8WKRqXkH9zSznbIXxtEcW0BmXbTnx6Eq23uKycijQkVUuQdbAj+VPrYfEWvZVrWtdZmQHfATWW/LnBE04h3Gql7osdRtqGz550n6lZ+X1a0QgbnpAz8VIm0pSqtlBveR2Dioy/no96RpyLzlhAjf90nwO3HJbE1+6Ux5GRgIcRX3KOBdpvQYtujZcwA9TBQU98BiccO2Spl9tVg/awnP1X+FDez+J6rsaDQVhiXQEISDnHQaziTWJ6Oc4GmUOuuXjMo81QrCe03iTHJR0MEju8Pb4wAsMZwlt1KjqLgUWfEF4ANmb3dWRniOb3PK4XJ4DCDQ4ffsbG7GX2sWNTFda6OG5uU4vyMun1PMSbWIxdpEY/WcNIkbfJqAXREhlEV4c0YAve6z59dDipLurFmmIglp3Se2hMyX/w0fWNRIMfsFBiVoxflnidbT7xOVMMHJHJKLD4aEi0mh2BNb1+kzJp5xf8Rao+jZ9Q4W4nOuvDBZEE6n2DuO0KZAcCK6ZXAASC9vs20p+PYEtegWe+/Amf/aU1EBHTHwdNEn67/rSB11V5jO23EstF8Zr/Gc0+/uC/iQUPrBNbd8acIWcUZsdNGjHWACwF7G3EAR61W6PsPJL0z+hek8/nOZp6/imF4T26UmDm9BtwK6JRyOdX+9ndAErj2bl2q5qV90g2xn/AneUOsRdKHn6xwHy1IuI9G8IcZETWJYbPkTwBMlOaDAgQ6K7QlRcEr75sDgeGl4pYIKkGaXZOqV1XrIGPTv4SjXSwFBe3zyribNi X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:15.8212 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 74d6b09f-7005-4c0c-a1e9-08dcd9c47fc5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00029928.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6367 To directly access the device memory, the HDM decoder registers on the path from CXL root port to the device needs to be configured when creating a CXL region. However, the physical HDM decoders are owned by the kernel CXL core when creating and configuring a CXL region. Thus the VM is forbidden to access and configure the phsyical HDM decoder registers. The HDM decoder register in the CXL component register group needs to be emulated. Emulate the HDM decoder registers in the vfio-cxl-core. Locate the BAR where the component registers sit. Take a snapshot of component registers before initialize the CXL device. Emulate the HDM decoder registers when VM access them from vfio_device_ops->{read, write}. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_cxl_core.c | 208 ++++++++++++++++++++++++++++++- include/linux/cxl_accel_pci.h | 6 + include/linux/vfio_pci_core.h | 5 + 3 files changed, 216 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c index 68a935515256..bbb968cb1b70 100644 --- a/drivers/vfio/pci/vfio_cxl_core.c +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -283,6 +283,90 @@ static const struct vfio_pci_regops vfio_cxl_regops = { .release = vfio_cxl_region_release, }; +static int find_bar(struct pci_dev *pdev, u64 *offset, int *bar, u64 size) +{ + u64 start, end, flags; + int index, i; + + for (i = 0; i < PCI_STD_NUM_BARS; i++) { + index = i + PCI_STD_RESOURCES; + flags = pci_resource_flags(pdev, index); + + start = pci_resource_start(pdev, index); + end = pci_resource_end(pdev, index); + + if (*offset >= start && *offset + size - 1 <= end) + break; + + if (flags & IORESOURCE_MEM_64) + i++; + } + + if (i == PCI_STD_NUM_BARS) + return -ENODEV; + + *offset = *offset - start; + *bar = index; + + return 0; +} + +static int find_comp_regs(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + u64 offset; + int ret, bar; + + ret = cxl_find_comp_regblock_offset(pdev, &offset); + if (ret) + return ret; + + ret = find_bar(pdev, &offset, &bar, SZ_64K); + if (ret) + return ret; + + cxl->comp_reg_bar = bar; + cxl->comp_reg_offset = offset; + cxl->comp_reg_size = SZ_64K; + return 0; +} + +static void clean_virt_comp_regs(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + kvfree(cxl->comp_reg_virt); +} + +static int setup_virt_comp_regs(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + struct pci_dev *pdev = core_dev->pdev; + u64 offset = cxl->comp_reg_offset; + int bar = cxl->comp_reg_bar; + u64 size = cxl->comp_reg_size; + void *regs; + unsigned int i; + + cxl->comp_reg_virt = kvzalloc(size, GFP_KERNEL); + if (!cxl->comp_reg_virt) + return -ENOMEM; + + regs = ioremap(pci_resource_start(pdev, bar) + offset, size); + if (!regs) { + kvfree(cxl->comp_reg_virt); + return -EFAULT; + } + + for (i = 0; i < size; i += 4) + *(u32 *)(cxl->comp_reg_virt + i) = readl(regs + i); + + iounmap(regs); + + return 0; +} + int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) { struct vfio_cxl *cxl = &core_dev->cxl; @@ -299,10 +383,18 @@ int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) if (!cxl->region.size) return -EINVAL; - ret = vfio_pci_core_enable(core_dev); + ret = find_comp_regs(core_dev); + if (ret) + return ret; + + ret = setup_virt_comp_regs(core_dev); if (ret) return ret; + ret = vfio_pci_core_enable(core_dev); + if (ret) + goto err_pci_core_enable; + ret = enable_cxl(core_dev, dvsec); if (ret) goto err_enable_cxl_device; @@ -324,6 +416,8 @@ int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) disable_cxl(core_dev); err_enable_cxl_device: vfio_pci_core_disable(core_dev); +err_pci_core_enable: + clean_virt_comp_regs(core_dev); return ret; } EXPORT_SYMBOL(vfio_cxl_core_enable); @@ -341,6 +435,7 @@ void vfio_cxl_core_close_device(struct vfio_device *vdev) disable_cxl(core_dev); vfio_pci_core_close_device(vdev); + clean_virt_comp_regs(core_dev); } EXPORT_SYMBOL(vfio_cxl_core_close_device); @@ -396,13 +491,102 @@ void vfio_cxl_core_set_driver_hdm_cap(struct vfio_pci_core_device *core_dev) } EXPORT_SYMBOL(vfio_cxl_core_set_driver_hdm_cap); +static bool is_hdm_regblock(struct vfio_cxl *cxl, u64 offset, size_t count) +{ + return offset >= cxl->hdm_reg_offset && + offset + count < cxl->hdm_reg_offset + + cxl->hdm_reg_size; +} + +static void write_hdm_decoder_global(void *virt, u64 offset, u32 v) +{ + if (offset == 0x4) + *(u32 *)(virt + offset) = v & GENMASK(1, 0); +} + +static void write_hdm_decoder_n(void *virt, u64 offset, u32 v) +{ + u32 cur, index; + + index = (offset - 0x10) / 0x20; + + /* HDM decoder registers are locked? */ + cur = *(u32 *)(virt + index * 0x20 + 0x20); + + if (cur & CXL_HDM_DECODER0_CTRL_LOCK && + cur & CXL_HDM_DECODER0_CTRL_COMMITTED) + return; + + /* emulate HDM_DECODER_CTRL. */ + if (offset == CXL_HDM_DECODER0_CTRL_OFFSET(index)) { + v &= ~CXL_HDM_DECODER0_CTRL_COMMIT_ERROR; + + /* commit/de-commit */ + if (v & CXL_HDM_DECODER0_CTRL_COMMIT) + v |= CXL_HDM_DECODER0_CTRL_COMMITTED; + else + v &= ~CXL_HDM_DECODER0_CTRL_COMMITTED; + } + *(u32 *)(virt + offset) = v; +} + +static ssize_t +emulate_hdm_regblock(struct vfio_device *vdev, char __user *buf, + size_t count, loff_t *ppos, bool write) +{ + struct vfio_pci_core_device *core_dev = + container_of(vdev, struct vfio_pci_core_device, vdev); + struct vfio_cxl *cxl = &core_dev->cxl; + u64 pos = *ppos & VFIO_PCI_OFFSET_MASK; + void *hdm_reg_virt; + u64 hdm_offset; + u32 v; + + hdm_offset = pos - cxl->hdm_reg_offset; + hdm_reg_virt = cxl->comp_reg_virt + + (cxl->hdm_reg_offset - cxl->comp_reg_offset); + + if (!write) { + v = *(u32 *)(hdm_reg_virt + hdm_offset); + + if (copy_to_user(buf, &v, 4)) + return -EFAULT; + } else { + if (copy_from_user(&v, buf, 4)) + return -EFAULT; + + if (hdm_offset < 0x10) + write_hdm_decoder_global(hdm_reg_virt, hdm_offset, v); + else + write_hdm_decoder_n(hdm_reg_virt, hdm_offset, v); + } + return count; +} + ssize_t vfio_cxl_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); + struct vfio_cxl *cxl = &vdev->cxl; + u64 pos = *ppos & VFIO_PCI_OFFSET_MASK; + unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos); + + if (!count) + return 0; + + if (index != cxl->comp_reg_bar) + return vfio_pci_rw(vdev, buf, count, ppos, false); + + if (WARN_ON_ONCE(!IS_ALIGNED(pos, 4) || count != 4)) + return -EINVAL; - return vfio_pci_rw(vdev, buf, count, ppos, false); + if (is_hdm_regblock(cxl, pos, count)) + return emulate_hdm_regblock(core_vdev, buf, count, + ppos, false); + else + return vfio_pci_rw(vdev, (char __user *)buf, count, + ppos, false); } EXPORT_SYMBOL_GPL(vfio_cxl_core_read); @@ -411,8 +595,26 @@ ssize_t vfio_cxl_core_write(struct vfio_device *core_vdev, const char __user *bu { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); + struct vfio_cxl *cxl = &vdev->cxl; + u64 pos = *ppos & VFIO_PCI_OFFSET_MASK; + unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos); + + if (!count) + return 0; + + if (index != cxl->comp_reg_bar) + return vfio_pci_rw(vdev, (char __user *)buf, count, ppos, + true); + + if (WARN_ON_ONCE(!IS_ALIGNED(pos, 4) || count != 4)) + return -EINVAL; - return vfio_pci_rw(vdev, (char __user *)buf, count, ppos, true); + if (is_hdm_regblock(cxl, pos, count)) + return emulate_hdm_regblock(core_vdev, (char __user *)buf, + count, ppos, true); + else + return vfio_pci_rw(vdev, (char __user *)buf, count, ppos, + true); } EXPORT_SYMBOL_GPL(vfio_cxl_core_write); diff --git a/include/linux/cxl_accel_pci.h b/include/linux/cxl_accel_pci.h index c337ae8797e6..090f60fb9a3f 100644 --- a/include/linux/cxl_accel_pci.h +++ b/include/linux/cxl_accel_pci.h @@ -20,4 +20,10 @@ #define CXL_DVSEC_RANGE_BASE_LOW(i) (0x24 + (i * 0x10)) #define CXL_DVSEC_MEM_BASE_LOW_MASK GENMASK(31, 28) +#define CXL_HDM_DECODER0_CTRL_OFFSET(i) (0x20 * (i) + 0x20) +#define CXL_HDM_DECODER0_CTRL_LOCK BIT(8) +#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9) +#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10) +#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11) + #endif diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 64ccdcdfa95e..9d295ca9382a 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -62,6 +62,11 @@ struct vfio_cxl { u8 caps; u64 dpa_size; + int comp_reg_bar; + u64 comp_reg_offset; + u64 comp_reg_size; + void *comp_reg_virt; + u32 hdm_count; u64 hdm_reg_offset; u64 hdm_reg_size; From patchwork Fri Sep 20 22:34:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808723 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2085.outbound.protection.outlook.com [40.107.95.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2EB618DF72; Fri, 20 Sep 2024 22:35:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871721; cv=fail; b=pjYgiwkiWg8QDMKSzFcmg5yxycgeZV8lZI4wzcIBB9inWSigxu4y3YASDi2Ce0CgVqMLT2keLzPLExqjoI+7yGn2aS0Dg7zBmVfTCFS6W4hy6eWz0qb9wxKic+aiHLmHWA+JFmRopBD2rQPq80qRIsWoNJmQU1conYDWuHuwQxo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871721; c=relaxed/simple; bh=D94POwOjTBGD9UujCYyZPQeSMSPqynqou5n6fAMNN+E=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=aHP9mcFFIz/TOGAxn/imirr/piGB0IbPSb9JLUEjdOP879DTSTgyl4NFYcK8dgh3kYiLW3VY9izLYOH0uRLD1n1Dtlh3+ZfQS83wp9PvYqy+oSSu4orjSU1ox0KwHmOMbbBuvvtE3mE5Wimdt6v+GEmgJOcHV4TmdMv0MI0OYj0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=HkB5I6AS; arc=fail smtp.client-ip=40.107.95.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="HkB5I6AS" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Wr6wpZdOdozkPtIAeagNp/tXrYNDbSJbGFaAI5qqJOGX3WnDCN+Mj0qLsMEC5pPoF3aJ9wRGl42Fxl4N8h0MCLmDwgy35uoFyIoQIEI39bX9rBwckBX2U5QJALESn370h6Wfew1krA4B3wzAxFOW2PhrepHMMu3gZ4meOniwSurzFemHJbskju0iVOsapYzOBqMn5QLR4SZkeup0juURIZ9byKNQCYOO8uCW9gnHYjAG+yjm7mPSsmlPRBAd5zhNtWw4HZngXf1k1JokH4BkcEKCassZIDBhSQnNCDHP+QIimhMNfqq3Mxbxf9MA5fR9AtIBiJkHRy30Ug9vLQMmag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gwpgZ9jEaMvBiYHOtREiWi21V5vUFrGcZIkH/uTpn50=; b=oNIo7dSPtlwkVJ6wi3wJhF/fIBam2Wdkw4CYiWw/m35TpFYmqS9d+fRe2Y38P9L2S+112ZtgZYsS9zi7A69qswtx8MpFBCZEL+HawQNT5Owaj4ayAWZhLr8CmHB9hjc/UtnpE7Ad4697IZBXgb2khnNtmNOS4xJS0yCJYy5hIGziSWx/lQvy/uc+14+nOk1MKXImkxp3hUKwxht4SrFgQOL0pJTv870pTFv30emDGW/3W38OFfG4TP7N239EZOVnvONjdEsEYdf4NlsSNAeNJzMx3J3iRddgQiTwkOHsspzHFrlVcLIkUijSlLoRuUrJ281bKKGSr38hMXi8VnZKug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gwpgZ9jEaMvBiYHOtREiWi21V5vUFrGcZIkH/uTpn50=; b=HkB5I6AShcp2pkwZ8YP0QQ/tD3W6yPs0fj+qH0MFTxCbizAv47remtVwBk3zX8ehLGSQGB21HJUGsA9SU8kw3UiV7fiHJ/vHYnirvMJ2St4IsJv1Gy0dJGhctEL3kyxb/34eINz5yOyoDdTSVmn82XJmAqEwmqNG66fJoCQy6eWyfDmppwjv3j0qME5NSNvEEZ87IApUZdb3e6LTg36bP6FlnJtAn17LRUoW+vor0OFm0kyiBMp3hMXOdzIZw1VjcNAzDMLUEVg261sewcngOE2UBEqtqI1uIcGQypBWtHmzeWQ1luBhdVEj8BhMQYKVyU3aI+O6Ba7owjh3e6kEJQ== Received: from DM6PR02CA0157.namprd02.prod.outlook.com (2603:10b6:5:332::24) by PH8PR12MB7230.namprd12.prod.outlook.com (2603:10b6:510:226::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Fri, 20 Sep 2024 22:35:15 +0000 Received: from CY4PEPF0000E9D2.namprd03.prod.outlook.com (2603:10b6:5:332:cafe::87) by DM6PR02CA0157.outlook.office365.com (2603:10b6:5:332::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.29 via Frontend Transport; Fri, 20 Sep 2024 22:35:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D2.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:14 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:03 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:02 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:01 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 09/13] vfio/pci: introduce CXL device awareness Date: Fri, 20 Sep 2024 15:34:42 -0700 Message-ID: <20240920223446.1908673-10-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D2:EE_|PH8PR12MB7230:EE_ X-MS-Office365-Filtering-Correlation-Id: 2f319e89-6e3f-4cd8-1080-08dcd9c47f1d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: jPI6YGLcRnzcp9C8LMhCwT0e0F+ZUw86Z+sweCLwuzZsN5zrA2moqVyF6B9nDM9gy/sNe3yjUaB9ro1mSMfCb0BJvhEDbKLqwhmmkROEStFB4IA5JVJEkNyU4zANatHjajnUaHag7xGQuJnG6V8dTbk6dHgms3l/s+N3vl0og1qw4Yj9D5qzqh9oZYar/Vi3xsOcXPHHEBagRq7wtOlwtnw+7YLA3UTmsR5L07l+PEKAtGCDhsaaSixXycGZVUpn9RGOi0uWJNUPZQY5o9sfHWpO83RWC9uXPj2yyowBPWBIkMc8aFZ4J0t8SJUqR48ov+Lntwo50xBbRg3JlU/nPjf1cAPEJIdISGU+Ek2z7tBKzrfAj40WHkemkdId3q9jIpPXUBHVbFsh+oPI2JpjNOgYcZk/iHlwsn6RYaK18c6yy0l/NJwEZT74Zi899rrB9GgzY+yDLzDmvjPv7dYUm7UMFa1Oa0EVmWGQV11bpcBe5ENOb1JsFj0Gkijga0OAl8PyPyxrsYyaP7Nt8fk/FqTchfL/ngsS8dBYkOuohBjJrFCvFnJNBxt94bh2K5aPyZdMSKiGje8//QIad6XufwYEvfURDzMJaaZ5EYzycRkFdL31NM3evq3sYnYYEWIWxY9SU5Vwe2Akyh1G7kn/GMX0HQQfaYxXqW5XVETCMkRHTeH4hySol7uuAFrK87GjFmH7QSKYFp1yt39JG1dIG1HGNTbMI4U4KbAAE++LhDPz9840Iyb0n8/GwvGodIJ96qa+inAqrT5wqchiha+/KhTexjBbMCUIy3d9SdbrUZm61jwAQJmjkhOcFSpcibQ54dY8CuZHeWdN708ggxbMfosG+tRVfzETJiopvagnYIpk2+Jb8HXcrhdx5ZXXheL53ZsxC1i01+S6AarYTYbgh/v+pDV2N7u24x8QDh7kI9cokmKM4a8TGiCBK1Qznog16BrzOTYXlckJsHq/wdjeNss7NII6ZQk45QoaCxbbfh2Yvcb3gU5/38t6AGXP66McmugylXsO9hj0ZUPOmbjpooIsv010mvjK1Uos4sN8CnoBF6axFWelOmJK216HIVHKJxFJp/WvOjNjXwTrBZSUkUGHAcoCwvQom+F0CjHTsMQgYk/Wu0Q+v6FkP7UR86VOReBnpT14pSVpq1vlRbgCbCwHDZz+uYePYi361QiHhA/lA7XYrJXbdPwj6gaB8mv0rR2bi4akZe/fc/dtjsRMNAZXOCPq5eupNd4l3kh12N96zCyHTodgW5TBpao44pEvd44w6WMRBLu/hF2pyM3mX+lwZ7u6XHlG7rKK12BCpWqKLhZXKGr3dFras5vUOWmu2V3PJqgJc5ZojhRO4HuQLgXhwhvNUggRtmDcNh5v3V1BVFOY2lxVkb1iqnztV1Su7ppzoHXjI+lKVY1tQyFMsJ+9T1qpj16QlmcJfTgEzWehw0Y3yhOLpuapya2R1oua X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:14.7790 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2f319e89-6e3f-4cd8-1080-08dcd9c47f1d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7230 CXL device programming interfaces are built upon PCI interfaces. Thus the vfio-pci-core can be leveraged to handle a CXL device. However, CXL device also has difference with PCI devicce: - No INTX support, only MSI/MSIX is supported. - Resest is one via CXL reset. FLR only resets CXL.io. Introduce the CXL device awareness to the vfio-pci-core. Expose a new VFIO device flags to the userspace to identify the VFIO device is a CXL device. Disable INTX support in the vfio-pci-core. Disable FLR reset for the CXL device as the kernel CXL core hasn't support CXL reset yet. Disable mmap support on the CXL MMIO BAR in vfio-pci-core. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_cxl_core.c | 8 ++++++ drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++----------- include/linux/vfio_pci_core.h | 2 ++ include/uapi/linux/vfio.h | 1 + 4 files changed, 39 insertions(+), 14 deletions(-) diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c index bbb968cb1b70..d8b51f8792a2 100644 --- a/drivers/vfio/pci/vfio_cxl_core.c +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -391,6 +391,8 @@ int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) if (ret) return ret; + vfio_pci_core_enable_cxl(core_dev); + ret = vfio_pci_core_enable(core_dev); if (ret) goto err_pci_core_enable; @@ -618,6 +620,12 @@ ssize_t vfio_cxl_core_write(struct vfio_device *core_vdev, const char __user *bu } EXPORT_SYMBOL_GPL(vfio_cxl_core_write); +void vfio_pci_core_enable_cxl(struct vfio_pci_core_device *core_dev) +{ + core_dev->has_cxl = true; +} +EXPORT_SYMBOL(vfio_pci_core_enable_cxl); + MODULE_LICENSE("GPL"); MODULE_AUTHOR(DRIVER_AUTHOR); MODULE_DESCRIPTION(DRIVER_DESC); diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 9373942f1acb..e0f23b538858 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -126,6 +126,9 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) if (!(res->flags & IORESOURCE_MEM)) goto no_mmap; + if (vdev->has_cxl && bar == vdev->cxl.comp_reg_bar) + goto no_mmap; + /* * The PCI core shouldn't set up a resource with a * type but zero size. But there may be bugs that @@ -487,10 +490,15 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) if (ret) goto out_power; - /* If reset fails because of the device lock, fail this path entirely */ - ret = pci_try_reset_function(pdev); - if (ret == -EAGAIN) - goto out_disable_device; + if (!vdev->has_cxl) { + /* If reset fails because of the device lock, fail this path entirely */ + ret = pci_try_reset_function(pdev); + if (ret == -EAGAIN) + goto out_disable_device; + } else { + /* CXL Reset is missing in CXL core. FLR only resets CXL.io path. */ + ret = -ENODEV; + } vdev->reset_works = !ret; pci_save_state(pdev); @@ -498,14 +506,17 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) if (!vdev->pci_saved_state) pci_dbg(pdev, "%s: Couldn't store saved state\n", __func__); - if (likely(!nointxmask)) { - if (vfio_pci_nointx(pdev)) { - pci_info(pdev, "Masking broken INTx support\n"); - vdev->nointx = true; - pci_intx(pdev, 0); - } else - vdev->pci_2_3 = pci_intx_mask_supported(pdev); - } + if (!vdev->has_cxl) { + if (likely(!nointxmask)) { + if (vfio_pci_nointx(pdev)) { + pci_info(pdev, "Masking broken INTx support\n"); + vdev->nointx = true; + pci_intx(pdev, 0); + } else + vdev->pci_2_3 = pci_intx_mask_supported(pdev); + } + } else + vdev->nointx = true; /* CXL device doesn't have INTX. */ pci_read_config_word(pdev, PCI_COMMAND, &cmd); if (vdev->pci_2_3 && (cmd & PCI_COMMAND_INTX_DISABLE)) { @@ -541,7 +552,6 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev) if (!vfio_vga_disabled() && vfio_pci_is_vga(pdev)) vdev->has_vga = true; - return 0; out_free_zdev: @@ -657,7 +667,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) * Disable INTx and MSI, presumably to avoid spurious interrupts * during reset. Stolen from pci_reset_function() */ - pci_write_config_word(pdev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE); + if (!vdev->nointx) + pci_write_config_word(pdev, PCI_COMMAND, PCI_COMMAND_INTX_DISABLE); /* * Try to get the locks ourselves to prevent a deadlock. The @@ -973,6 +984,9 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, if (vdev->reset_works) info.flags |= VFIO_DEVICE_FLAGS_RESET; + if (vdev->has_cxl) + info.flags |= VFIO_DEVICE_FLAGS_CXL; + info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions; info.num_irqs = VFIO_PCI_NUM_IRQS; diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index 9d295ca9382a..e5646aad3eb3 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -113,6 +113,7 @@ struct vfio_pci_core_device { bool needs_pm_restore:1; bool pm_intx_masked:1; bool pm_runtime_engaged:1; + bool has_cxl:1; struct pci_saved_state *pci_saved_state; struct pci_saved_state *pm_save; int ioeventfds_nr; @@ -208,5 +209,6 @@ ssize_t vfio_cxl_core_read(struct vfio_device *core_vdev, char __user *buf, size_t count, loff_t *ppos); ssize_t vfio_cxl_core_write(struct vfio_device *core_vdev, const char __user *buf, size_t count, loff_t *ppos); +void vfio_pci_core_enable_cxl(struct vfio_pci_core_device *core_dev); #endif /* VFIO_PCI_CORE_H */ diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 71f766c29060..0895183feaac 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -214,6 +214,7 @@ struct vfio_device_info { #define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6) /* vfio-fsl-mc device */ #define VFIO_DEVICE_FLAGS_CAPS (1 << 7) /* Info supports caps */ #define VFIO_DEVICE_FLAGS_CDX (1 << 8) /* vfio-cdx device */ +#define VFIO_DEVICE_FLAGS_CXL (1 << 9) /* Device supports CXL support */ __u32 num_regions; /* Max region index + 1 */ __u32 num_irqs; /* Max IRQ index + 1 */ __u32 cap_offset; /* Offset within info struct of first cap */ From patchwork Fri Sep 20 22:34:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808726 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2055.outbound.protection.outlook.com [40.107.243.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 352EC18CC00; Fri, 20 Sep 2024 22:35:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871724; cv=fail; b=UfMUwtWJghtsv/QMUB7b63AIYZhn0mCTpgVpUq4HJxZqrzXSGnWFKMcPdEtF8O7cYJI+n6kyjbfWvNLd/17ZMji7q3BVwvI6O8zGUGJWFJZeUYzkv0nvpgeVOqPI0Vp4lu7+rtg6sN41z1EQYWGwpFuflSgeXyytRxCXUt3OobY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871724; c=relaxed/simple; bh=qOIXDRWqrqQFaJlv8C7IjYxLbl/RtTo70JRPoz7AhYw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mu3hKC64RggH2wzlSEvP1cCj1wupLkOxSoaXVzWP/UGUJc+nsvbakEdDSo6tb8B3XzM6BgmaDFqWL+aXyCBReYpOew/je1O+s7zohd3H2ZwYrui8mSqH23gZAyKbMXugJf4oRM2IgP8JPALTIHWxK5kewPAJZLTN3iH8tG/krtY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=iFtSvEun; arc=fail smtp.client-ip=40.107.243.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="iFtSvEun" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EkY3h31HBIoH/s5Tdt2Rl6QY14jRIdaMzJHdRQhDxFSK7AwOh8mrfYFEUKWhhIFnA3m47wTeD2uHG8AnekL7ncsa94dZwgFzm/eJJhejgSB90gRaO0f5WZEhmSHjelBi3xzldPQH1OxFjy/LZqz/Tome0fi2yCo6H+i02hN7m2UeVE2bekmcHu5+NpEdQiPrLjJEI/g6vdTJ3/ilNEwObfnMS8loCBjE8E1SDQ+dUqrv4J1kvyn9ww2UC0X79KiUFvKyXQmZdFenVZ6CG2/dqAyCbtUuJUotvl6woRE2i/kQGX48MbQGnemkh83f6VmpbbuzxZ4IIqpz79Gdtap1gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AOZg+9wRzHQZcXEu+6gWwHdy1JYiCQwiZ2RQzNPHX8A=; b=F04XtLU8TmzmUHWqKLh+yczmRdiYxcSG/ZP/kFjSi2CuEi/QHZrWHoV6d79WHE1/YoDnNwJf71A/IFFxuq9sL9UuhME7moghKtGhkhHwl8VBPo3nS1cGdnGUDvwoCbnvzn8phD39WwuzgZM5v9bfgTbS71c7+l+wA7J22kbmafNDII9A4nllSkn61OQMyjDuXK86u1lnabIKuUON1jhnsiIw62QTxopGyuRE/ZRgrJBMN3EJdtHv9nAknCFjt2VTqt2hR5Jh6/sGhYV14fQlgknoowIPEvvbezk9Pbuo65mFjKuqyo/XC3/Nsq4W1NykgjmDu/D63XRZ9t847oc7tQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AOZg+9wRzHQZcXEu+6gWwHdy1JYiCQwiZ2RQzNPHX8A=; b=iFtSvEunzPlBUZVo1vRwgDKG2SIutyXm+VMNiI4ONhWvSZp6rBqKunokuAJERWvoygulnSZTtx1PcI5kwTJOAOk7PCGe23QZu5niLTWsOJQlqelE8UPwM2ZDj5iAqxXpDmdRhdizyb1VHZO7n68HRYvQiCHGuvbAuPFuh0mBuny5Cy4Xpl7K45LlAGjpcjecG28CP1oklZVMaEz4KOKCe4NS2YdORa/U6skWmL9rAcPVyaWe45glm/ErhcSF2LKq67nhI+XyeNf0bp4S3ctgTLSNk1gRWfs2XRwh0Sj0L6xYciKOfohsh8lStLpnJ/DgKgxJ52vA1vNnRzbd3NSFxg== Received: from BN9PR03CA0685.namprd03.prod.outlook.com (2603:10b6:408:10e::30) by SJ0PR12MB5635.namprd12.prod.outlook.com (2603:10b6:a03:42a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.24; Fri, 20 Sep 2024 22:35:20 +0000 Received: from BL02EPF0002992A.namprd02.prod.outlook.com (2603:10b6:408:10e:cafe::17) by BN9PR03CA0685.outlook.office365.com (2603:10b6:408:10e::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.25 via Frontend Transport; Fri, 20 Sep 2024 22:35:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF0002992A.mail.protection.outlook.com (10.167.249.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:19 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:04 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:03 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:03 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 10/13] vfio/pci: emulate CXL DVSEC registers in the configuration space Date: Fri, 20 Sep 2024 15:34:43 -0700 Message-ID: <20240920223446.1908673-11-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0002992A:EE_|SJ0PR12MB5635:EE_ X-MS-Office365-Filtering-Correlation-Id: 8945d7a3-2ca6-4203-a575-08dcd9c481f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: HvPNNXsCRKTJowPcuNjdzej/pg72+JdpeQG+PsaQYifTh4D1FntVrw9I/DG4h3XX/HPv4cYLilcZgj/ZJIXDfVo7KnM9AzWcXnqpfXI8DvJmnW5Skd6VRcNwNG8xeQfjSqyhuUytYAAyTxmLkE0ZIhNLT1AM3F6Gbb4FmPZgjsZobaMyYaXHs6azjJ1KilT//4iGHMQ7fHrWAm5lHbmMWVqgwAwaUvBrDeSg0NyeQt1y9Nr8pFq6aisoT5NIh7hF90BKicVnGoYRgk4Ch0dRY4P/2Kqwnn0vHdur4iuc091O4Ywdtg4TKcqtoIlcltr4zcNNT87kbZa0fQNHkICECDPYiqI0HpUvICTNQrpcW7qwnxtZtSaeG68IC+kg/74qnMeTLu3dS5NasA1/KgMd75GLPolvEabKZpFDwq5EvLUWnHpt7ZQYN8qTLfd0armTgcDZjdS5FtBzj45z8TYG2kkKXZ9v2mundOtWnWbOXp4vuTU0900qgOGIvib9aLugxORru6zXgEHkA2c8iP0djXitdis6q6164gS+bHALcmKmek6eDWv8HH80VtEPxxn+f9+9tBa7Kdg2NSr4HYsctlObs1SgoWRshBVg5VNUN3Y2KTaAU4p/Iosd58LhHliv/AIVJQ9dpB/EF+9bV74+dExR+/vronRCFPpqVpkKRQyujpuOOjGxU1pQ+qkdP/jpkmiLXmF4ixqki8RC02tXN9aS0VUiaRJomkMvsvzOhx+cCz2Ny26dfVi8gV3rYVWiL4vN6xrccDZ0UDlYLGQ/druyTo/iUzswBu/seqMrY4iSGpftq/uadjY5t8JWQ1hFSg4pqrjYdL7scAOjZIYGwxjm9lb6xSWjX5wELASabWkOluJOVqZbbglK+G7ENHSlecvnrsH+Tb3uB02R7fvt35SDQSZsaSTp12G1xFIYjX7J+ouDgmZ8Raglr44PF+t5HUANyn6naN5flqrCxiwiAHf/4yw/zSSIIzIBW0W8N1dSYkHgnAm6Cn4oBErtN/huzxoI6DkwEijz9PmSgKU5bqnBI07pjO0m0edlxeJrR4HkKbQtMJNGKTkzqIvpogf0UorOnayufyzMvJzZamRgCwAGlbFEnvJG2dtxCj15zlAn2KYBX33nt9gbd6+yTXhzt9PIZRNMrWlAsmy45kw8gAL3MSusExQH5MRKZ8YSXmVQ33BC1ujKjcIuFWu1/H2P1t/AN8QOrw0wP0anM8jOpG6b3kVkY9KUCbf2gQWn/hy7aKPE7+Wxhf5F8oRBf82Jz31JAv9tRcxiiFD74yAlasYLJbml48rwALSBf5JjkqDqMb49G2a118Dy+NGYStF0EMLjPAoB+D1sBd63Le5lP4voyUGwP0KEZwb7AprzX110tMbmenPzaTKKOuPi2tya X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:19.5513 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8945d7a3-2ca6-4203-a575-08dcd9c481f7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5635 A CXL device has many DVSEC registers in the configuration space for device control and enumeration. E.g. enable CXL.mem/CXL.cahce. However, the kernel CXL core owns those registers to control the device. Thus, the VM is forbidden to touch the physical device control registers. Read/write the CXL DVSEC from/to the virt configuration space. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_pci_config.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index 98f3ac2d305c..af8c0997c796 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -1902,6 +1902,15 @@ static ssize_t vfio_config_do_rw(struct vfio_pci_core_device *vdev, char __user perm = &ecap_perms[cap_id]; cap_start = vfio_find_cap_start(vdev, *ppos); + + if (cap_id == PCI_EXT_CAP_ID_DVSEC) { + u32 dword; + + memcpy(&dword, vdev->vconfig + cap_start + PCI_DVSEC_HEADER1, 4); + + if (PCI_DVSEC_HEADER1_VID(dword) == PCI_VENDOR_ID_CXL) + perm = &virt_perms; + } } else { WARN_ON(cap_id > PCI_CAP_ID_MAX); From patchwork Fri Sep 20 22:34:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808727 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2056.outbound.protection.outlook.com [40.107.94.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D312218CC02; Fri, 20 Sep 2024 22:35:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871729; cv=fail; b=sBaTZcyyHHz6+fsXHHM1wfZwd9RjaGp8h4wl2MTauSnGwCdhdZ0eW/IDfV8jo/oswwLDX0EBdJ+DIX+oAXYgJM1vpxOzp1j3tMn4ncv2aGfFiDutSdyZpECrvAjGSgEBxBUttjXzRRjzK3xFY19/gr0qB8cxYihXBxuaxo9+1qk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871729; c=relaxed/simple; bh=HzZQT4JTb9Vjz2bFDT6w1c2OsIzone4ffRvzbROFczw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mY4Ml5BQFILQ1WtpOJ5mzpVb0hV3BncaSYZJ7FkAwpf9HHLvEb1PxLJV8/7VV2Nar7/Rynjr/xjRM7r0AJac/Sui1z/9lqP+qRFoRVZYmKh1F3Ep+KoiuLyTd0o6ufMPGJuLBAWtskhhKjZ/4L9peh3dXZae3TQ8W6jP/GCceyM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Bo80OZxb; arc=fail smtp.client-ip=40.107.94.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Bo80OZxb" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RwGoWL0opU291yM/9yipfYm375mVhQMjLXz1jmLRs7c7aF0LUK4L6UEbQOur0qUGxmZqK5zLvA3pTHhZKKQNewjyXvOgp4fL8y9X1YIpdgE1OCuh4mTVMATF8BT9R6GQo8KtMb2FhWSxUktaWc/k74txvrN1BPM5Y3shCfZl5HZRHuA2Md7GQpN4NynWVPka7JRD8M+1a/RL3QeXx20JHP/hKqLfibBO2UZZ7vZ+sbGTRNdQDaRmz9JFTpXNw5Bh6ylWGXrOHR699cI00LfRDMUXoInN9thuX2LcYo97UUD6BlDxLB70WEDV6kWV6hM7q4M8kyK07w5DTy+xNUGXmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j31hJvif7Wk95e95Osg2cHJ/eYP8VQ4zfkemBrkNQWw=; b=WybAVIzdj9tS3l5FqkYR6u/HdvEiGqXtLaiQZ7n2z7onVFKB+rnyf+dCGDNzw1fblGZbNRB0S0XtGO+/0BKpTdlHEpAT2z7Rw0UExa+6ZGAR9yScIhu26hiy7QLA+1f/QPhOgK5wkHErfBZ2xAxcW8/zqGVDU4FBfQ6J+1KIIXc2kDygECIyVT5D5ElifeHqffuANtYmF7inw1CoOIsQo4GYA8PkoB+AjP0nhMdpz5dvjs7p3coMHn9vYC6PGWLALAFarICDlUdKGm0DZJzrznhmFCRney8QFb3zlW5irmYWr4e3INTGCK460rzuGt44Y6xSW+k5hIdgua4nDoudRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j31hJvif7Wk95e95Osg2cHJ/eYP8VQ4zfkemBrkNQWw=; b=Bo80OZxbBrHaeYEsKfqroVWJz9nc3i5MBJ/CGD9qcV6qjHWs0Bld8RZvQioWDfGPs/uzLQJjvlUpsOLhNqvJ9dyZLucbeezdBvYk3mmHsB7seAndOhh3nns8IpKN/2jB8q654rpH7+L1HoHsGrFYAFl4eaWhcLabflD1dZRyNJ+fZGI20JgTpL/VPMFYWLMfD/umRssUC8PXC38K1tJD5LCEzGNwuvmeGoXKEeg1Ax0/fp5CNMFqO3gkPhiAwlePbE35D9yJuaKTn3zjWEHNzvRfHlGDZq6dAAIBWza6HQfZAhQUFMU1w6RjFSDW4O6yGzMQR/9WesZbNXjpY8/X5Q== Received: from DS0PR17CA0005.namprd17.prod.outlook.com (2603:10b6:8:191::6) by DS7PR12MB5719.namprd12.prod.outlook.com (2603:10b6:8:72::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.17; Fri, 20 Sep 2024 22:35:23 +0000 Received: from CY4PEPF0000E9CE.namprd03.prod.outlook.com (2603:10b6:8:191:cafe::59) by DS0PR17CA0005.outlook.office365.com (2603:10b6:8:191::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.14 via Frontend Transport; Fri, 20 Sep 2024 22:35:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9CE.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:20 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:05 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:05 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:04 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 11/13] vfio/cxl: introduce VFIO CXL device cap Date: Fri, 20 Sep 2024 15:34:44 -0700 Message-ID: <20240920223446.1908673-12-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9CE:EE_|DS7PR12MB5719:EE_ X-MS-Office365-Filtering-Correlation-Id: 9058ca05-4d2c-453e-6d7f-08dcd9c482cb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: uhVArqJlAfYE07kVGJdqRSTnnRHv6892S37uoxmRQJtIr7thzlNlXPTL9Yii7s8u7UGMTZPQ05bjWHYAF9TYp9myb0BH+A/gCWTS2CxzBPRo0pphxghzroLeXDK767ZWiL0BQZAdYZhg8bvrVryDguICbUlB5S/EPDnQBlDvjgq7ogsQpN8PYdg182vbbOQI0PLKBkD4tJO+LzU3ih5s2EZKvmNCGyMz/YDzYY8srwJKiQOyNhKtxisNPQETmp0sM9RU5e+z0LfhXVUQFaCEF2KI1IXNeqDoGoKbQYag2Nh6pCXetSOdM1RPiZsfjoPM2vXgWGCMj/7lW4mgaCTkIFyLl3/bnze79OAfOz6n2sqOpOYxGs57u8z+V3TV2YAkH93GT3Qz4aVgMrGtutX7OZ2zogPhe39EnQXBB+fehzFdtvxstHsvYs9Diult3cNQMjGfQ1zZlyfOG+YOFSJQowlBGI/OeCOxzE1Z6kuxPXdP2mq7HymhQqDkD9M+q74OSLs9LqYC8q5nyA0OrO1H0TQwhMe6OHZp3tXG8Uu44/oEebDDjyCpDi5Diu7Kp4/K6qkEm/u5QnD7bJA1mkQV9b1VKQobTYm+veVgFcE66s16TGCKTAjMei967Fj6KrFUTsPKpJOKYZzQ1Z638MMOJGtbdgUMJxOvIR6p433mNo721fN7qH16saO0Ycxo2lCHpwJWPbhHEJW9/MHconFJEcUMjTapJPlXXI1Qnc6EQCvIc8M9pM6a/3ARSSlnr0oFG6LwFzP9DVMAPkCto5hv1RxCsKfO8mCwDtzD/NaDI6X2zXeBfEf71TJ6uydDt4AChA2CIVdsg1o2dFtJJyIy4pleywNgz/Gc/ihyKHCt2JLrdg9GTzadWnMdb9Zcj4YebgcAMkEo3zEgq1gyXucBE7pTthwfcRq3BduFEXdQl69OPT/Lw1xtCF52aJ3A9Xm+i0+p74wO45+QBJp1K1zu8j7Sj4Gn/Hd3btP7ItWI7ha4s7IA2tXYfFvLtnFvwpgpWjtUldqMzElM2B1iEhbDLGHhL77d/0cwGqxh7E4bgueGdzsXK4rBjogzjWRsSSJSPY/KbAzQeHoOs9IbZRD4JBdLzj0c/G1G70uAb0EOUfxCwwnUPuzT1uDMzmatxZPZWNIU24Uc1QlQxPRZOPm3wKm+B9vDmGOs9VQHzR2uSVOWVrnW7ruLDm/8ewJeB3Ufjjz3z65k3BGTmtr3DiolZP5ERZrvOOOL/Q0Vs/9hvM2d93aON87Qf/D4L9Z+Qx9oH6SBP3hIlgEaoPn+i2HdwuC2B4ORNRC822B1Pu+uBonmjhtWxj+IJEFeAeRzYpF/9ddPDoRqKNkMXVY1BBl1cqJTSii6DLzjpY1cit19WXfDN6urN4mdyB/CaHUTTByPA9Eds/giNtJuQWZTedyVQsuulYVrXQqzKOuybtwlMAuS79NPWDvK7hn1EA0jSDtW X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:20.7830 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9058ca05-4d2c-453e-6d7f-08dcd9c482cb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9CE.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5719 The userspace needs CXL device information, e.g. HDM decoder registers offset to know when the VM updates the HDM decoder and re-build the mapping between GPA in the virtual HDM decoder base registers and the HPA of the CXL region created by the vfio-cxl-core when initialize the CXL device. To acheive this, a new VFIO CXL device cap is required to convey those information to the usersapce. Introduce a new VFIO CXL device cap to expose necessary information to the userspace. Initialize the cap with the information filled when the CXL device is being initialized. vfio-pci-core fills the CXL cap into the caps returned to userapce when CXL is enabled. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_cxl_core.c | 15 +++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 19 ++++++++++++++++++- include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 10 ++++++++++ 4 files changed, 44 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/vfio_cxl_core.c b/drivers/vfio/pci/vfio_cxl_core.c index d8b51f8792a2..cebc444b54b7 100644 --- a/drivers/vfio/pci/vfio_cxl_core.c +++ b/drivers/vfio/pci/vfio_cxl_core.c @@ -367,6 +367,19 @@ static int setup_virt_comp_regs(struct vfio_pci_core_device *core_dev) return 0; } +static void init_vfio_cxl_cap(struct vfio_pci_core_device *core_dev) +{ + struct vfio_cxl *cxl = &core_dev->cxl; + + cxl->cap.header.id = VFIO_DEVICE_INFO_CAP_CXL; + cxl->cap.header.version = 1; + cxl->cap.hdm_count = cxl->hdm_count; + cxl->cap.hdm_reg_offset = cxl->hdm_reg_offset; + cxl->cap.hdm_reg_size = cxl->hdm_reg_size; + cxl->cap.hdm_reg_bar_index = cxl->comp_reg_bar; + cxl->cap.dpa_size = cxl->dpa_size; +} + int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) { struct vfio_cxl *cxl = &core_dev->cxl; @@ -401,6 +414,8 @@ int vfio_cxl_core_enable(struct vfio_pci_core_device *core_dev) if (ret) goto err_enable_cxl_device; + init_vfio_cxl_cap(core_dev); + flags = VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE | VFIO_REGION_INFO_FLAG_MMAP; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index e0f23b538858..47e65e28a42b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -963,6 +963,15 @@ static int vfio_pci_info_atomic_cap(struct vfio_pci_core_device *vdev, return vfio_info_add_capability(caps, &cap.header, sizeof(cap)); } +static int vfio_pci_info_cxl_cap(struct vfio_pci_core_device *vdev, + struct vfio_info_cap *caps) +{ + struct vfio_cxl *cxl = &vdev->cxl; + + return vfio_info_add_capability(caps, &cxl->cap.header, + sizeof(cxl->cap)); +} + static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, struct vfio_device_info __user *arg) { @@ -984,9 +993,17 @@ static int vfio_pci_ioctl_get_info(struct vfio_pci_core_device *vdev, if (vdev->reset_works) info.flags |= VFIO_DEVICE_FLAGS_RESET; - if (vdev->has_cxl) + if (vdev->has_cxl) { info.flags |= VFIO_DEVICE_FLAGS_CXL; + ret = vfio_pci_info_cxl_cap(vdev, &caps); + if (ret) { + pci_warn(vdev->pdev, + "Failed to setup CXL capabilities\n"); + return ret; + } + } + info.num_regions = VFIO_PCI_NUM_REGIONS + vdev->num_regions; info.num_irqs = VFIO_PCI_NUM_IRQS; diff --git a/include/linux/vfio_pci_core.h b/include/linux/vfio_pci_core.h index e5646aad3eb3..d79f7a91d977 100644 --- a/include/linux/vfio_pci_core.h +++ b/include/linux/vfio_pci_core.h @@ -80,6 +80,7 @@ struct vfio_cxl { struct resource ram_res; struct vfio_cxl_region region; + struct vfio_device_info_cap_cxl cap; }; struct vfio_pci_core_device { diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0895183feaac..9a5972961280 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -257,6 +257,16 @@ struct vfio_device_info_cap_pci_atomic_comp { __u32 reserved; }; +#define VFIO_DEVICE_INFO_CAP_CXL 6 +struct vfio_device_info_cap_cxl { + struct vfio_info_cap_header header; + __u8 hdm_count; + __u8 hdm_reg_bar_index; + __u64 hdm_reg_size; + __u64 hdm_reg_offset; + __u64 dpa_size; +}; + /** * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, * struct vfio_region_info) From patchwork Fri Sep 20 22:34:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808729 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2074.outbound.protection.outlook.com [40.107.236.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7A1C18E347; Fri, 20 Sep 2024 22:35:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871732; cv=fail; b=YRTpHC8fEsFwC90UeGLtizOUBbVF5I27AjhgMtn4QnOcIDImCwj4Z7SWLjZyITWC4FLJAlMfFb/DyPoPrpSps1+fCChOlciC78klnzz1JRe29QuSU8LYANdwQLho898USeSMoDowJbMVAsU4PjlcHGFmPD0C8rDI0DGK9M4heu8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871732; c=relaxed/simple; bh=KSVOTVWhQ7FOBawrx6yHcRPfDw6BIR/lcjz14rPP/d0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=O1Pm22VTlmF0bao/jHFNtcOC40ZgY0E2nEISc55HMgGNiagupljdFo/J87qPvSTKrZi37W76qkGa9OcLhY1L7gBwCbfTvrn5EjWUoMcw6OGjoOgYBSLu9EYugJVmrhfRGjwj54FXbw/jTpgqGqyFeMhB8gJv2njr9vWiDxeJhB4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=aJfKteDs; arc=fail smtp.client-ip=40.107.236.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="aJfKteDs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZvPoCa0OPf9iRiP+eTpqvQuulL62+vPROeVxBB80jnqcFHYgLQDhYI39SXWL0TJBK6SexuFIxfeIT7rqk9iFEV3kjSTXUdooMhcScW6YXvOeaG59QghcPnXp4aQXqVZyoJ0Eq6drZG2v7DDKWwPeAaAKdHYPmNxZk1/BKzchOdoHRnetU2LG8peg3yu5lc1Flit+2GUO7WrpiTS5K3NdZ5pJyWY6gA60Jrn/F+JYqC7eW+/0obViMurrleyo0jSVZJv/Tktq/vfF611Ee9roDAaRwC8oleBIn4UsQT29iyI+fGtP2D35SDFJt71Q1mdBD/S0vVgn84uX6R9VsmzYEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Jj6vIJHUIuEGQEoT6g/SGfa4v6kHpkW+QLMs33NhB6w=; b=bxhqgAU2f+b9gOd2JzP5Fc/2tR3LO8XX0pRVPDpxATzLv4+3tafhuaUoYIBP4xPkmoQMJF+BT1nc70izUYfEF/yGkq21iiXmCBI2ITaFOYuo38ZcoVHfRmr9lEkP5ClT/nKS4nyPenxjrzP3+WLsAxFGhBGUacUSDPrbMFAPijd24yfIs4MJi+NxcNGvd4LljXiw57A9gyQiojGeJCYmc8izxuR3ZxmyY8UPRLLDx1feHHhdJs0BzeO4jpFFsCr2LzjrRDAS47MzURZezKOiQ3NGY87VQ+xc2Zfwp7EzNy6G/BwuYE4HhU9fm/2amrPoufGFZMclGdHwZyE9xBJpNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Jj6vIJHUIuEGQEoT6g/SGfa4v6kHpkW+QLMs33NhB6w=; b=aJfKteDsN9bS2lXhLEdkjndT0NLl/yhtjtjp+yNXWTjBUmELWLpTX/V5L6NqVxF86AZjnLqJvhmzeniGBH7Ulhc6i2BJ1tWNdXR+l9ZzG56QxI1DFZsInr5om8LLOlqIWSA03KwQyUxHpbSyQnS7AwCZk5IqdlvV4JZ2yGScBxi3+sCbErzgHhL3HgTgVF9fvn7G/QbsZ1GwmhUBA2riDF7ve6C8E+tPdL7qlfOhoRM527dJTUe9QTzbFEvYNTyxyfuskztN6jRh6B1oAtvVFrMxGWIlgNFRJxoGVkKAR3dgppo73LMXnLcKTo5PfEJdKUIuHbC/2ZBpo47AHblbyQ== Received: from CY5PR19CA0119.namprd19.prod.outlook.com (2603:10b6:930:64::13) by MN0PR12MB5788.namprd12.prod.outlook.com (2603:10b6:208:377::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.22; Fri, 20 Sep 2024 22:35:23 +0000 Received: from CY4PEPF0000E9D3.namprd03.prod.outlook.com (2603:10b6:930:64:cafe::6c) by CY5PR19CA0119.outlook.office365.com (2603:10b6:930:64::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.30 via Frontend Transport; Fri, 20 Sep 2024 22:35:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CY4PEPF0000E9D3.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:23 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:06 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:06 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:05 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 12/13] vfio/cxl: VFIO variant driver for QEMU CXL accel device Date: Fri, 20 Sep 2024 15:34:45 -0700 Message-ID: <20240920223446.1908673-13-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D3:EE_|MN0PR12MB5788:EE_ X-MS-Office365-Filtering-Correlation-Id: b602c0b5-eaa5-4c8f-3421-08dcd9c48464 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: UnuJ4auzK0KCUzTxLahkH4nblPEkaAQwevxzn3jJ6pKocSeCicS/KQqVqlHNu7NxX+L+eFM5mhesUqGl7sk19clkOSspDB6dWd1JesWZ2Au6PjicKIgAwiPycPMzjLLDnLMf3mtPnZyXOsPnamvki+JFKyEVduqVsWehmo9NcLKzTVpUJqYhnXRTtZcY0ITaeFK6MrJCa0b50Lp+q/lyz0j3louNaRoWPPs/F2NEwlDH9/kAVcaN6FcX64EQfqh5vp+FDlZFIzoEFmx9AJ81BRWLhqN2yzJFF3O9Sj7L6WfUFkFB0D6r3S5SLmZOjsfNd3cCXiEdYIFeYzdQ1ygPXnfPdZvuqBwVze8aV5jaMFmp4UrIWxFM4Enxzcgl5/v6XLome63Oj49chVMfJK7azSlGhBMbLAzhtXTiyytacJuK5rHmXSd0FbRv+Z+CswYqaSOE9Ih00ujt8vHE6/ygJeKq9folAfRai3ok99Q6ipx9Eyoz1miVLEpZCy8sXGVlhXe1Guug+cK1CH4vxvMz2KvlGa2jDtPNTHj2oYIlhSUA+C3L0JKXwcvm3d7owHKoMgZkWcCGEhlpadSKrW+kouIF77RIoxeieTZ4BRuOnNX+NxO2M0KhsHFgxz2BOX114mTjFAV/LhYFkjh07mqSqfUZn7YcMvwrKDlxlZDb/5JJ06iO35mSYIfmqWehlYE4o3PGsUx/jpMhjBWUTjwKYvGu3Rzsrm4vf48xSOIEs74XBmDj4fc/MsQufhO0rg/5VemNMOSgEatjDlYW0rWxR/T0KbMW+qMOe4GFZN0P9e/BgPzgl2C+Y0/JJvMqnnXZJG/3mQjOoshlU8kBeISu7IGG5x9KbbPYW4x+Y6DMcvKxu1STp06v/bblnJ6nYU6tJK54cfpjm1bcmAMDZUExifv0srGtkSUuQilIeZ1GN6q/RS5oADTgpyDnK9rvjZhGo9jEJHHIHjMsstr5nIl4pyvUSpUA9badviyoVS1QtjOw26AXgFR0YT6gd+ZMHWwCWXSPCFjXJkqKMsevRiSCMEvtPQsg9vss/pDkZoQo53B02ainTn+Ku4k4htmNOY33uWVpaMVT/H4pwyIQtt2tHdeBk/J2cpvb3kRp4WcL8VGXENprCYiviPL94OcBGJaBsvcJsexvG/+QKNqCPJVsbjoi2BnqWY5rm4KHtoIO5CqWB+pgwxKRp4KB+4eoBh1mqXjpV/L05Xz+ckqf3S+nlqmrlNxTFHZESy9hAm+qHDxigd1Yfax5kdBiqiOUDKuSSSK27/T2XkVLDykJv3C/HI19TmA1KJYQsuHTuPWzaKU+6jxdn2wcp4Ou1Yr/OpR+SxZ6o4ZCOs31jb6//v50DbN6wwX4u+x7zgs5hYDrgw+vCXxWv1S7lQEjo3fFm8SD+JOwdh8kNvMefWan2dpVFQfd6IuQH3AGXPgDUIAbzSpr29abhtgk2x+y9Z627RqN X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:23.6338 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b602c0b5-eaa5-4c8f-3421-08dcd9c48464 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D3.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5788 To demostrate the VFIO CXL core, a VFIO variant driver for QEMU CXL accel device is introduced, so that people to test can try the patches. This patch is not meant to be merged. Signed-off-by: Zhi Wang --- drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/cxl-accel/Kconfig | 6 ++ drivers/vfio/pci/cxl-accel/Makefile | 3 + drivers/vfio/pci/cxl-accel/main.c | 116 ++++++++++++++++++++++++++++ 5 files changed, 129 insertions(+) create mode 100644 drivers/vfio/pci/cxl-accel/Kconfig create mode 100644 drivers/vfio/pci/cxl-accel/Makefile create mode 100644 drivers/vfio/pci/cxl-accel/main.c diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig index 2196e79b132b..9eebce09ffa2 100644 --- a/drivers/vfio/pci/Kconfig +++ b/drivers/vfio/pci/Kconfig @@ -75,4 +75,6 @@ source "drivers/vfio/pci/nvgrace-gpu/Kconfig" source "drivers/vfio/pci/qat/Kconfig" +source "drivers/vfio/pci/cxl-accel/Kconfig" + endmenu diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index b51221b94b0b..03293b52c5e3 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -22,3 +22,5 @@ obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/ obj-$(CONFIG_QAT_VFIO_PCI) += qat/ + +obj-$(CONFIG_CXL_ACCEL_VFIO_PCI) += cxl-accel/ diff --git a/drivers/vfio/pci/cxl-accel/Kconfig b/drivers/vfio/pci/cxl-accel/Kconfig new file mode 100644 index 000000000000..c3c9d7ec7fa4 --- /dev/null +++ b/drivers/vfio/pci/cxl-accel/Kconfig @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only +config CXL_ACCEL_VFIO_PCI + tristate "VFIO support for the QEMU CXL accel device" + select VFIO_CXL_CORE + help + If you don't know what to do here, say N. diff --git a/drivers/vfio/pci/cxl-accel/Makefile b/drivers/vfio/pci/cxl-accel/Makefile new file mode 100644 index 000000000000..20f190482cc9 --- /dev/null +++ b/drivers/vfio/pci/cxl-accel/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_CXL_ACCEL_VFIO_PCI) += cxl-accel-vfio-pci.o +cxl-accel-vfio-pci-y := main.o diff --git a/drivers/vfio/pci/cxl-accel/main.c b/drivers/vfio/pci/cxl-accel/main.c new file mode 100644 index 000000000000..1672fb9d9232 --- /dev/null +++ b/drivers/vfio/pci/cxl-accel/main.c @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved + */ + +#include +#include + +struct cxl_device { + struct vfio_pci_core_device core_device; +}; + +static int cxl_open_device(struct vfio_device *vdev) +{ + struct vfio_pci_core_device *core_dev = + container_of(vdev, struct vfio_pci_core_device, vdev); + struct resource res; + int ret; + + /* Provide the device infomation to the kernel CXL core.*/ + /* Device DPA */ + res = DEFINE_RES_MEM(0, SZ_256M); + vfio_cxl_core_set_resource(core_dev, res, CXL_ACCEL_RES_DPA); + + /* Device RAM */ + res = DEFINE_RES_MEM_NAMED(0, SZ_256M, "ram"); + vfio_cxl_core_set_resource(core_dev, res, CXL_ACCEL_RES_RAM); + + /* The expected size of the CXL region to be created */ + vfio_cxl_core_set_region_size(core_dev, SZ_256M); + vfio_cxl_core_set_driver_hdm_cap(core_dev); + + /* Initailize the CXL device and enable the vfio-pci-core */ + ret = vfio_cxl_core_enable(core_dev); + if (ret) + return ret; + + vfio_cxl_core_finish_enable(core_dev); + + return 0; +} + +static const struct vfio_device_ops cxl_core_ops = { + .name = "cxl-vfio-pci", + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = cxl_open_device, + .close_device = vfio_cxl_core_close_device, + .ioctl = vfio_pci_core_ioctl, + .device_feature = vfio_pci_core_ioctl_feature, + .read = vfio_cxl_core_read, + .write = vfio_cxl_core_write, + .mmap = vfio_pci_core_mmap, + .request = vfio_pci_core_request, + .match = vfio_pci_core_match, + .bind_iommufd = vfio_iommufd_physical_bind, + .unbind_iommufd = vfio_iommufd_physical_unbind, + .attach_ioas = vfio_iommufd_physical_attach_ioas, + .detach_ioas = vfio_iommufd_physical_detach_ioas, +}; + +static int cxl_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + const struct vfio_device_ops *ops = &cxl_core_ops; + struct cxl_device *cxl_device; + int ret; + + cxl_device = vfio_alloc_device(cxl_device, core_device.vdev, + &pdev->dev, ops); + if (IS_ERR(cxl_device)) + return PTR_ERR(cxl_device); + + dev_set_drvdata(&pdev->dev, &cxl_device->core_device); + + ret = vfio_pci_core_register_device(&cxl_device->core_device); + if (ret) + goto out_put_vdev; + + return ret; + +out_put_vdev: + vfio_put_device(&cxl_device->core_device.vdev); + return ret; +} + +static void cxl_remove(struct pci_dev *pdev) +{ + struct vfio_pci_core_device *core_device = dev_get_drvdata(&pdev->dev); + + vfio_pci_core_unregister_device(core_device); + vfio_put_device(&core_device->vdev); +} + +static const struct pci_device_id cxl_vfio_pci_table[] = { + { PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_INTEL, 0xd94) }, + {} +}; + +MODULE_DEVICE_TABLE(pci, cxl_vfio_pci_table); + +static struct pci_driver cxl_vfio_pci_driver = { + .name = KBUILD_MODNAME, + .id_table = cxl_vfio_pci_table, + .probe = cxl_probe, + .remove = cxl_remove, + .err_handler = &vfio_pci_core_err_handlers, + .driver_managed_dma = true, +}; + +module_pci_driver(cxl_vfio_pci_driver); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Zhi Wang "); +MODULE_DESCRIPTION("VFIO variant driver for QEMU CXL accel device"); +MODULE_IMPORT_NS(CXL); From patchwork Fri Sep 20 22:34:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13808728 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2063.outbound.protection.outlook.com [40.107.92.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7504618CC00; Fri, 20 Sep 2024 22:35:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871731; cv=fail; b=cq9fKDTDLnwG1PNO9vjwTOHLi+dZH5smN5pKWvAA+3xdVatN+920TL8o6OIIk1SaTAI2As88lRjfmcirsKGv/ZEiaP8j9PqkGvzb+QdL581oVFtCErNIRmO3ZLo0WY8ZDQA2zRxwilu9U25MI3iUR9Wh24O/QoVMvqB2fdc0xO4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726871731; c=relaxed/simple; bh=S9seR/d0SVyLFTzT31jY9HV1U2jInepA0Tm4eCjUG8k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=sMUAyiiD2nqYnhmsSV1/nSWj0IFCdwImD9VMqb5N/1apM16lBSSMZoJZLcxHqtd74t4hg0iyxfwsifwKB86np1GmUsYsYWy1Fnt2Do56vpDvbzTRdinl8aE03k5D1i6FPN8T8JJTOf8WntVDsRdg5Ck0AkvSbN5Z5lrbbc6YeZs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=KIYz1VNm; arc=fail smtp.client-ip=40.107.92.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="KIYz1VNm" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=v2M8KsSMJtk8Kd+sICVEelvvH89ks0BFHJ3wE9mxuzJoyFfZh7Yo6/yPxd5x6evd0AlOTslPvA3AGHEire12Xkp6ljdKrObFcRafIiRJSL2XF+Z5FlyJff+99W/imknH1+UESYCnZgXessvVPk+pV94eBoyiuIBknpEyPRJT5F/BYIjqZCtM2KBvJUWWqxkH25ePsKqMkPIZ+QhoCCJHT6w2GWzbK1nl2d5z2OHTgY4GU5wYBnBUewhZZJE/gJ74Desp/Cf2HHu1dOhvV7QkDlfvL/TbTjkzYAjsuGu3K/DFx6nNhNp3mCLoLpIOusSq0GM58pdOwX7iZ3CC5U5a1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eGUDSrJQe9VV7/IpqIXRQO7ulPicSS7l9Z3LgxEvO7o=; b=iXiEaEagujTff4Cv/uAQCJjc1jvCGgdeXDOAbtebkAcAXySgdoWfTFfMpJj2uNXZDH0CR9gk+aS/5UiCR1YeO+xS07omwylL/mvdcj4G8BMqMpfXn5frmD3lxORPzMwbnX9ZX5Wa8gyFu0MY6/DwWOxkIMrhjuTfdBOWChlQG1F0O1lpvCKgtZPVEsiblmWiyf07kMLeeaA+LWXRaa6v8x6KAmYN2GM0I8IOgYPo9w8wwqmnB2O/CiHZkcFvShw7lfB2vIk5PMMpy/MRAWt3kTld1yO9DAwflGj8Pd4dZKGqurqJQfOTqFvfBRHKQdcngjuj+0iLjTyTzhtOd4XbFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eGUDSrJQe9VV7/IpqIXRQO7ulPicSS7l9Z3LgxEvO7o=; b=KIYz1VNmi99dGUBMHwcGf4/1JQ86eJYLM5WZOQHZ1qWPHVjepEtG+KecKaWuzu4Mu3A3AADcHPPeDyBFN81kEyg8FNKZeXPEpxvSWinOoaj6PL6+/tm7E4CsbqG+oPPUIbaI4PNlNygPWBaiohD3jpEeD8Nplk74porUfALGsaIH4usvNbg2FGzGT71ABN2463v2nqF34IouBKEfzZKpQP+S6Vk+tTr62QFH+tZXDXBZcoAhoz1HBQ78zsXXkhj5bfjwLUwPRvaXKzCpcARmrlqW0YYaQ6vPBvTV4eSzifN6Eprk0aSAkaUg+80ipIyXFzbvqJqNFeAjLkgo/kr32A== Received: from BN9PR03CA0688.namprd03.prod.outlook.com (2603:10b6:408:10e::33) by SN7PR12MB7452.namprd12.prod.outlook.com (2603:10b6:806:299::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.21; Fri, 20 Sep 2024 22:35:24 +0000 Received: from BL02EPF0002992A.namprd02.prod.outlook.com (2603:10b6:408:10e:cafe::a) by BN9PR03CA0688.outlook.office365.com (2603:10b6:408:10e::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.29 via Frontend Transport; Fri, 20 Sep 2024 22:35:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by BL02EPF0002992A.mail.protection.outlook.com (10.167.249.55) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Fri, 20 Sep 2024 22:35:24 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:08 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 20 Sep 2024 15:35:07 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Fri, 20 Sep 2024 15:35:06 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , , , , , , Subject: [RFC 13/13] vfio/cxl: workaround: don't take resource region when cxl is enabled. Date: Fri, 20 Sep 2024 15:34:46 -0700 Message-ID: <20240920223446.1908673-14-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240920223446.1908673-1-zhiw@nvidia.com> References: <20240920223446.1908673-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: linux-cxl@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0002992A:EE_|SN7PR12MB7452:EE_ X-MS-Office365-Filtering-Correlation-Id: e5f1de34-add2-4b63-f95b-08dcd9c484f4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: x1oK/IPxaSt6TbWzlRCmcv7HmXJO37cZsmpJWYkvk1QJmyr4m5mykTbp5mXkYID+JLhU3pfY3dbW4JUVcN211+ytJFwFPPMNMlZNzZsdxwRvQPmk+ECXQp1FAD0LEsGdmCDIR6NT9xxhyABq4TcqstpzSlMS8Qqk6a1QRlHZgqEhpfHr+lHa09tYdTxq1Yd5NWg/56MQozFsN76Q2hajikqGR24Y5Vz2+9mig7qGDkBVOLmt+70G6s82zpmy1qAJQroUtFMDjKo1kGJu//RAuA0fBCD1GBqkGcqgmYxidAPXzqxpYh1Zp9/s4qo6YMWa/XF3d5e7XuNYSRSKlLbMicTZtolBNlGFt2KsAu+qN8T7XZ+9tRGRH1MpBW/Ai1xUeMoLu0EW8PQxdOBzyPWXG51H2wY8mfy54f+bMLFrP/95XW+XtCdJAxjKPPc8pxm1bZAvCdJw+tcElXhzIbS/PeFx4M6j3wSPFeEwGQknIWNLEXq6PKXJio4Kr84Kznc5S+5rsURt6zurXOLeayyOuITDSuvHrF8OEH4qs9UWV164WVE+ZtqKQsLbb4YLF0wobPjBZ3tY3N1FVOl/5cqu3uXmlY2Dtw85Ulmtt0oGoE2cVs5oedWtKXoySNH/CFL4YhahV+yPEYWEo0I+aHO3xEwAmEiE0d6RtTCjvoVaTXzu9UuJfcfPYy9bpxyl5Ft/En62p3GiGB9jGUhnrrEM6rHrYtOAqUep8zanD68xFCRGVaZSc0Y0SaV9aoBQv+e5VyO3wAlLgEWSJu4JJS0chIWUCCo4C8rhG9VJ64nkrMG7C5UIH/0JU91o/OW96Ctu+ATSSCJDt4Qt/spXMKauokT4yuhygqKpD0nOJzhzz6bP3BSqLPuWjAxZIsq4jMCy0tYCSY50fatqhOcSXvWEcWPDoWi/QTUZMxS0b4MJPVHDG2lngSEmARq8OBCW7WH7nXItSideraBVqebQfANw/xEYbvhv2H54GsveqA8dgbXOR6yTGZPYFTsigpq8qifqckU6JAKKzoiVWLXia91hkZhM03U8S48P9XMfzWpVsTNibOwBLVzTVYmcv8XPTPrIK+UNKQgk3Ff9ElBu/2Vv2afyX+ZJXWoPSWzxXWsWari2zk5AKuIYEorOC2sM0u5mymFKJgELjz505AOPanNW+Io2AN4lz88bZBhyGABdYZ/QA6J1IZwzFJcvS+pmRsO0vKeM5uxy2BVYQrumLp+gmWUHDOxaMn2KCw9mJSCJyOQQv86vdzbnGmAnby/M4qmxSLhsrX79cenrKpi4Wz8LVF9aAKl4OoElC3X6WWiKhZuBMNAFpBR5sRXz2VxkAuwViGM5QPzyrFtJzz1BX70NAhJkzUDHvriHSApXZSyn7CiosieIE+/7t55A+LzW2oswNl5UHGEE1H/KXv/kGVs3c0kYcXzq9Qid46j9G2RluQCpmFDCbiG5PewKvs8bQo6F X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2024 22:35:24.5044 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e5f1de34-add2-4b63-f95b-08dcd9c484f4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0002992A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7452 Looking for a better suggestion here. vfio-cxl-core uses the kernel CXL core to initialize the CXL device and the kernel CXL core has requested the resource regions when accessing the PCI BARs. Thus, requesting resource region in vfio-pci-core always fails. Signed-off-by: Zhi Wang --- drivers/vfio/pci/vfio_pci_core.c | 15 +++++++++------ drivers/vfio/pci/vfio_pci_rdwr.c | 8 +++++--- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 47e65e28a42b..91f8b984b53c 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -633,7 +633,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev) if (!vdev->barmap[bar]) continue; pci_iounmap(pdev, vdev->barmap[bar]); - pci_release_selected_regions(pdev, 1 << bar); + if (!vdev->has_cxl) + pci_release_selected_regions(pdev, 1 << bar); vdev->barmap[bar] = NULL; } @@ -1775,13 +1776,15 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma * we need to request the region and the barmap tracks that. */ if (!vdev->barmap[index]) { - ret = pci_request_selected_regions(pdev, - 1 << index, "vfio-pci"); - if (ret) - return ret; + if (!vdev->has_cxl) { + ret = pci_request_selected_regions(pdev, + 1 << index, "vfio-pci"); + if (ret) + return ret; + } vdev->barmap[index] = pci_iomap(pdev, index, 0); - if (!vdev->barmap[index]) { + if (!vdev->barmap[index] && !vdev->has_cxl) { pci_release_selected_regions(pdev, 1 << index); return -ENOMEM; } diff --git a/drivers/vfio/pci/vfio_pci_rdwr.c b/drivers/vfio/pci/vfio_pci_rdwr.c index 66b72c289284..df7b5aa078e9 100644 --- a/drivers/vfio/pci/vfio_pci_rdwr.c +++ b/drivers/vfio/pci/vfio_pci_rdwr.c @@ -214,9 +214,11 @@ int vfio_pci_core_setup_barmap(struct vfio_pci_core_device *vdev, int bar) if (vdev->barmap[bar]) return 0; - ret = pci_request_selected_regions(pdev, 1 << bar, "vfio"); - if (ret) - return ret; + if (!vdev->has_cxl) { + ret = pci_request_selected_regions(pdev, 1 << bar, "vfio"); + if (ret) + return ret; + } io = pci_iomap(pdev, bar, 0); if (!io) {