From patchwork Sun Sep 22 12:49:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809093 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2049.outbound.protection.outlook.com [40.107.237.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 189A5645 for ; Sun, 22 Sep 2024 12:50:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; cv=fail; b=LOqtke/OjQRYbbRfcp7VABBjrDJNB6Iq8kjBiUydPFLsV4+RTbmeFO5eh3jMgkEC/O5wx0TbATPy7VFvgRE0U1KRmMjOTpkpsthHXqxqRFr9fV8OxFA4Mit/3qBWVEk4fWPQSPwmf3fetutcpiawa5ufPcfysHWVXjvoH+O/mCQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; c=relaxed/simple; bh=N3PD2eEjIn2yzOXwiid2f/sbIF1lMCn0jPjQINEKnJc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oHpzrYzCX817tf7G3F9PqZT2vsrazgu0n9v4YubzG+UydBybjnAs9FYT6OxLyMH0LKMBIKtqE6fjHyMyfmLvQvUSaeeCWxJD2woDukJ4hSBu497Z7Skgg79SRkhwXb5vc27jxGaMX68Lhl5sG5jjAW/vp9ufYnbcWvAwmGlCOEc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=eSXsuVZf; arc=fail smtp.client-ip=40.107.237.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="eSXsuVZf" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iQSdVEJfIyH5CRh8bnXU5Rxit+yncuNaVO16UdybgYHfgJ/W1ffYi3G/bGSVuSpRL7eD0cMKuUAMr7Y2/Vuf0SB7Q/FY81gEIopTIvJuZXCzdE71c18V+MIXAhiFcY9/+OfxA7CkyzXr4Jq7YgOD/GC+9iaxP0nuKSFWv/ntqAUHnIKThIqRLBTNguAX7BZxf7Z7Wc5aKHKSwBCv+zQNGXxae5rYWcETbh71C3VKP33sv4M0RyNRow2kJZs8iu1reR2PE9jUUusKVSv4omQZk4Guwpj2VDBSzqQ54kG06LdvWuJg2CVB/mGY2sicLX8/zn33iE1kgc0K7O5uM4QyzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Nx273nmDeUobuIlmdXJNLUGMxhf++1hApE+yWwM2oPM=; b=m1+39mqAWd37tnJRuKpfuMa4DRuthwHP918ABrZCq4rYStS8CZQ21Bozad5nUCvUgYEfyVJBb5aVEr9MrJXuYVPZrndm/N2zl5Q8eo17vGVugA60lvOkDVm+Q2QVMcH3vi97adaEQT/Cocgm52CHSU+8cZMxgRfWcPQ/Z8lhvsSnd4NwBO6DnchGBgGsw2B7ndH//leuydN8lEYwW7dpGTrpTxP6JahHj8kBskMn9UdZwW8BLrXwGV3pqnfnd+DEIMOE51MLm21fT9G/pioAoHkOyKd0ac6GyhDI7n5k0z1I0DmjBdJ05xrK6NBGG6kDjp5apAobgLof8Bs1fB3a+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Nx273nmDeUobuIlmdXJNLUGMxhf++1hApE+yWwM2oPM=; b=eSXsuVZf8rnHCgb8AayNTh5B2ipvOq0yOY9vNUFZiI0cBfaS2P+rSP1raaT/yNIMoObE5uS0gFre+u5WKUj4MMne4zbQIjYN1pJ4839EumDQMa5kbuNJ2ls1lfwaXNNOuCA/tb7+1oKl/s74ZfhSYpyQNkLkx4DSgw1mH5O3/ubFmh9fwp784naU6PLL+7SoiXKdO8YpbR/AX/TgqPhXIq7EHp5zhjiwuG8eAC/Cck7XVtNA7wPSNgfMg/yf7d+pPGlK9vcw4B3WbqettEoPSqe1PlLbS5/lniCw/hW4mEzb0i8O+Y0mw2HDSlN/nCs89eD+Y0XQoBtJwgvPerkQlw== Received: from BN9PR03CA0283.namprd03.prod.outlook.com (2603:10b6:408:f5::18) by SA1PR12MB6775.namprd12.prod.outlook.com (2603:10b6:806:25a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:29 +0000 Received: from BL6PEPF00020E66.namprd04.prod.outlook.com (2603:10b6:408:f5:cafe::73) by BN9PR03CA0283.outlook.office365.com (2603:10b6:408:f5::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.27 via Frontend Transport; Sun, 22 Sep 2024 12:50:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E66.mail.protection.outlook.com (10.167.249.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:29 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:20 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:19 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:19 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 01/29] nvkm/vgpu: introduce NVIDIA vGPU support prelude Date: Sun, 22 Sep 2024 05:49:23 -0700 Message-ID: <20240922124951.1946072-2-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E66:EE_|SA1PR12MB6775:EE_ X-MS-Office365-Filtering-Correlation-Id: 108d6cd6-6a13-4f75-3757-08dcdb05234c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: snWZ8AbsFsdEGxqV6rkQSqySAyQ4q42yXs0zPj21flskujrIW0SyWJIjRhDAygvn3xwtODLfk6WNhyrWG0EHr3NCOZTyWr5zpp++WixqlL1iwI1DYmZtA/X0a4faWxLeGH6s/pOCHbH30P4+RWyjP6qRm9NrH+rrvhZsLNHxEX5UsrVTYQIKoPavOnUTbCLwUSHqYUOHbJ5FWMavw0OWmHRVjSiVuoMhWAyoB6MpsKEHPXXZRt6qqK2Hic+C6IjuJ5UrVDCNy6kXyhYpd86gE4c/d5P+cEo3WSR0XzUKs+AhP/pymQAtTR6SCgLuzkCkVnLLhIHHKLQVaQyF8KbUW0335ks0mSizNpR2Zk7el7TUMUZHQKr7nvlyXws8Rssm/lSRNoRij0iA/zY67A3mQfJRcHU6SPnbxSI/xy8D4cIqsSACQjuxZ0K4aVHHYn4v2UljsFx1ZMUAVDGOoFjc8h3/zjKIolwAPVCj1qpV4gybJCozNLLitsz5o86/DEjJOYNpQvh63RdDXYS+wuBOhbPErILYFiBgunLVu0Y3JrGxG86ijy+/LO8HHljFOuBeDFivDIZF8+eVkPAK9rXfap16rf8c7PN9Bsifo+KZmQBQQLOk6ePIVkRCp7pwz0rt3J9CYBLegbdLgUnnma0uGKLjPPQuAhNt05EPdsoT/pDEuyPYxq+AFGx7VeSAlNTdofjM+cGySIvoqXXgBvgyBumJPRGrTsKqENZA8hyWUhJKgcXQSB9/0kFZSagwpdboBFwLHDgYE+s1XwMLluis3NwNoMndRuO9heomuFvHRGmKtPB+OtuNfxpY6OHlDioH3J6krJ5c9rtBDyD9Edt2/sbupLvjSEAvzr4a+bvNfT8CXs95eWuqpiyiXyQVCPqj3vxbpNbq3ZQgh0hYuvFbNmb3ttlPC/yl0OYCgjTZ56/hPuYSx1wEcTdAN3imqGgJkyKhPjPYeTAQdA0UGPMaUc5ox1TFBYmGEqoNA2xo4XJGZmBRE1/F7m1EmK7vGmsr5CKwyf8BOoVRqOLkMol0+++n4g4pN4/Vk1r408tyjD74I20fn4sRXi/8iYXreevpHfdFfEJn80xyLHHL+L5ahoENqOwwUfuzLPdmzl3XXEhcKKUlw+T/WkyKlolEGuwNH2V8lI+I5JjsXTUxzh4Liow8L0EtHEQ+yFf6s8IEsPswqxAGkK6iv8+1NsD75tPnPvjR8kEGmFrsLNofzyySBiUWT/Km0MdM3LsTmalCVyX0bKAwnZpy8yb7T2xcCDrAIM+EBGrmUa8PPCb4AA4bHi4Qj10bi/GTIbQnlRbC+7XBFsfL1Zfaep3I8ofRxB1s56OBrC0RFQaVzZN26CFIBBiJOaw14kkMtmLZDtGTk/eTQnljf9NigG9AQV28i0pGSiTrDvw3ZugFa37xlTSyHORx4ioyTA/93yLE8pXMzTUiK2Cpj/WoJweYOd0OEtEc X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:29.1409 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 108d6cd6-6a13-4f75-3757-08dcdb05234c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E66.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6775 NVIDIA GPU virtualization is a technology that allows multiple virtual machines (VMs) to share the power of a single GPU, enabling greater flexibility, efficiency, and cost-effectiveness in data centers and cloud environments. The first step of supporting NVIDIA vGPU in nvkm is to introduce the necessary vGPU data structures and functions to hook into the (de)initialization path of nvkm. Introduce NVIDIA vGPU data structures and functions hooking into the the (de)initialization path of nvkm and support the following patches. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- .../drm/nouveau/include/nvkm/core/device.h | 3 + .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 17 +++++ drivers/gpu/drm/nouveau/nvkm/Kbuild | 1 + drivers/gpu/drm/nouveau/nvkm/device/pci.c | 19 +++-- drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild | 2 + .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 76 +++++++++++++++++++ 6 files changed, 112 insertions(+), 6 deletions(-) create mode 100644 drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h create mode 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild create mode 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c diff --git a/drivers/gpu/drm/nouveau/include/nvkm/core/device.h b/drivers/gpu/drm/nouveau/include/nvkm/core/device.h index fef8ca74968d..497c52f51593 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/core/device.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/core/device.h @@ -3,6 +3,7 @@ #define __NVKM_DEVICE_H__ #include #include +#include enum nvkm_subdev_type; #include @@ -80,6 +81,8 @@ struct nvkm_device { bool legacy_done; } intr; + struct nvkm_vgpu_mgr vgpu_mgr; + struct auxiliary_device auxdev; const struct nvif_driver_func *driver; }; diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h new file mode 100644 index 000000000000..3163fff1085b --- /dev/null +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef __NVKM_VGPU_MGR_H__ +#define __NVKM_VGPU_MGR_H__ + +#define NVIDIA_MAX_VGPUS 2 + +struct nvkm_vgpu_mgr { + bool enabled; + struct nvkm_device *nvkm_dev; +}; + +bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); +bool nvkm_vgpu_mgr_is_enabled(struct nvkm_device *device); +int nvkm_vgpu_mgr_init(struct nvkm_device *device); +void nvkm_vgpu_mgr_fini(struct nvkm_device *device); + +#endif diff --git a/drivers/gpu/drm/nouveau/nvkm/Kbuild b/drivers/gpu/drm/nouveau/nvkm/Kbuild index 9e1a6ab937e1..d310467487c1 100644 --- a/drivers/gpu/drm/nouveau/nvkm/Kbuild +++ b/drivers/gpu/drm/nouveau/nvkm/Kbuild @@ -8,3 +8,4 @@ include $(src)/nvkm/device/Kbuild include $(src)/nvkm/falcon/Kbuild include $(src)/nvkm/subdev/Kbuild include $(src)/nvkm/engine/Kbuild +include $(src)/nvkm/vgpu_mgr/Kbuild diff --git a/drivers/gpu/drm/nouveau/nvkm/device/pci.c b/drivers/gpu/drm/nouveau/nvkm/device/pci.c index b8d2125a9f59..1543902b20e9 100644 --- a/drivers/gpu/drm/nouveau/nvkm/device/pci.c +++ b/drivers/gpu/drm/nouveau/nvkm/device/pci.c @@ -1688,6 +1688,9 @@ nvkm_device_pci_remove(struct pci_dev *dev) { struct nvkm_device *device = pci_get_drvdata(dev); + if (nvkm_vgpu_mgr_is_enabled(device)) + nvkm_vgpu_mgr_fini(device); + if (device->runpm) { pm_runtime_get_sync(device->dev); pm_runtime_forbid(device->dev); @@ -1835,12 +1838,6 @@ nvkm_device_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) } quirk_broken_nv_runpm(pdev); -done: - if (ret) { - nvkm_device_del(&device); - return ret; - } - pci_set_drvdata(pci_dev, &pdev->device); if (nvkm_runpm) { @@ -1852,12 +1849,22 @@ nvkm_device_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) } } + if (nvkm_vgpu_mgr_is_supported(device)) { + ret = nvkm_vgpu_mgr_init(&pdev->device); + if (ret) + goto done; + } + if (device->runpm) { pm_runtime_allow(device->dev); pm_runtime_put(device->dev); } return 0; + +done: + nvkm_device_del(&device); + return ret; } static struct pci_device_id diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild new file mode 100644 index 000000000000..244e967d4edc --- /dev/null +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: MIT +nvkm-y += nvkm/vgpu_mgr/vgpu_mgr.o diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c new file mode 100644 index 000000000000..a506414e5ba2 --- /dev/null +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: MIT */ +#include +#include +#include + +static bool support_vgpu_mgr = false; +module_param_named(support_vgpu_mgr, support_vgpu_mgr, bool, 0400); + +static inline struct pci_dev *nvkm_to_pdev(struct nvkm_device *device) +{ + struct nvkm_device_pci *pci = container_of(device, typeof(*pci), + device); + + return pci->pdev; +} + +/** + * nvkm_vgpu_mgr_is_supported - check if a platform support vGPU + * @device: the nvkm_device pointer + * + * Returns: true on supported platform which is newer than ADA Lovelace + * with SRIOV support. + */ +bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device) +{ + struct pci_dev *pdev = nvkm_to_pdev(device); + + if (!support_vgpu_mgr) + return false; + + return device->card_type == AD100 && pci_sriov_get_totalvfs(pdev); +} + +/** + * nvkm_vgpu_mgr_is_enabled - check if vGPU support is enabled on a PF + * @device: the nvkm_device pointer + * + * Returns: true if vGPU enabled. + */ +bool nvkm_vgpu_mgr_is_enabled(struct nvkm_device *device) +{ + return device->vgpu_mgr.enabled; +} + +/** + * nvkm_vgpu_mgr_init - Initialize the vGPU manager support + * @device: the nvkm_device pointer + * + * Returns: 0 on success, -ENODEV on platforms that are not supported. + */ +int nvkm_vgpu_mgr_init(struct nvkm_device *device) +{ + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + if (!nvkm_vgpu_mgr_is_supported(device)) + return -ENODEV; + + vgpu_mgr->nvkm_dev = device; + vgpu_mgr->enabled = true; + + pci_info(nvkm_to_pdev(device), + "NVIDIA vGPU mananger support is enabled.\n"); + + return 0; +} + +/** + * nvkm_vgpu_mgr_fini - De-initialize the vGPU manager support + * @device: the nvkm_device pointer + */ +void nvkm_vgpu_mgr_fini(struct nvkm_device *device) +{ + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + vgpu_mgr->enabled = false; +} From patchwork Sun Sep 22 12:49:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809089 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2071.outbound.protection.outlook.com [40.107.223.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4466B28FF for ; Sun, 22 Sep 2024 12:50:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.71 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009437; cv=fail; b=WSXCAhjE7WeAscYSiu40x3zvH+huBpRYrTHRRDInTy1IrktHElxI9hVQm/iap8P81c3yaUrSfFgruWj3X7v7SZTKfpk5FdlSGwEwHpzCU/kKbnlk6022nDk/RFYRKxRL485cv1Y2hccZrYYMTj20hFWV5Sf9oh+HicAUhdkZsxk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009437; c=relaxed/simple; bh=H2k5GFO7e4iPzatK17kbyHShCdsLpjuoKiV321Xv/64=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ZjC7GcKkZifsFWt3QezVA3uTZ9q/70p2m5ZrRGDhBri21rcteC4NBsRcjPX3ulHpziF1icXXrGDpdaWugdgSJwOZqbjgA7F3AaXVYMXACNHi/2266WftSJKXfPPP7NfJSYngC1oaSfd7SICxtDOgkCFwin5bBsWQDrOBfG35PGg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=KM4xXN97; arc=fail smtp.client-ip=40.107.223.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="KM4xXN97" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lpmKufB0CQOnTrjz0hs/IvlOFOYOhbxTQUL/niP1k50m//LjWoR+fHOS6hKYEZoM8/gk2ACwu4KQGnZW2xR7EFmG+prVlv56xhLYHJQAXGvPNC+zz+KdJsYCJvpQCR3K4POLlM33OqQpaDBlEh586bcNboQd6JtI9mpT6nal0Gs5Gbu4duJWrXebQ1S8gaIYOoUzpCb9oblL45xgKls3BahS+Tjui1Xsu+3+JKNExM2ycyfSbLa25ULRvFcPugK7VEsko8RyHXZabN2sI8hnYkMM5teMk6GuAK4ukRpt0swGzxb2GQKza8ll85eJhP4CxjK8nvr15z9n6WYHuKzI7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JhYlEorIr6rz6xDIW90zTcFP3NMnHIUCs/fKt3Piago=; b=iKrFe/VI116bzdwfaWCTHMU1fvaH2GRqBMqXmUTxUc8SwGu4ApwpoblfmcMdFy+Z2GQnutvcfQhahvKMkRAwTed3GYbtZndRJXNPgO+SmYlSxXVdrcVl8roX23dLIIP1XnUairkA9IUlNzD097RAUscmlK0MTC5joYZIhCArhQa+mmPuY3L9NOcpKeEeI0QJmi3XvMnVdWKahesUbJXlRracFOZEe8CYpV6s+zeHTe224DcKVzqkinIQDtbtNTjH+0lnmEmDecbtP2xTfLYe36sR7Mf2nF1dzLLWVY/V7g9bPhGk8udSK4dWOjkyjxPNoasdsZZzAHkA/17OXxQubQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JhYlEorIr6rz6xDIW90zTcFP3NMnHIUCs/fKt3Piago=; b=KM4xXN97SaPJKFUJg6n/FLyL4pcDoTGT3tiZjLffK4NCkj7KBsd5J/0IQ/xDSOZBy7H7gbE3ZKFkwM+Md6HIU0/0Jw69pwT2PS9W1nNrOI78bppHfiEW9BMGlhZfkyUibxsvJMzjWt8V6PTSpqGfsoVoywnmWoZ0dBC3sNcq3ggoE29/z8Jwfg+danRtRv7ydOCLr4m91vV30O7/9z6crPptNCzB7IVooAXID7J1F+rLjH5ejbF5bVYewld52IxMPr7P0pKBb5j4I8iJs6gA+2uzilGiHSVHo1ZNY3gpfEtZUgjGQ8Sf+csRJ6RpnyptLc8YjrT+kmTEy0gL37ufZw== Received: from BN9PR03CA0282.namprd03.prod.outlook.com (2603:10b6:408:f5::17) by DM4PR12MB8452.namprd12.prod.outlook.com (2603:10b6:8:184::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:30 +0000 Received: from BL6PEPF00020E66.namprd04.prod.outlook.com (2603:10b6:408:f5:cafe::c3) by BN9PR03CA0282.outlook.office365.com (2603:10b6:408:f5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E66.mail.protection.outlook.com (10.167.249.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:29 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:20 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:20 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:19 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 02/29] nvkm/vgpu: attach to nvkm as a nvkm client Date: Sun, 22 Sep 2024 05:49:24 -0700 Message-ID: <20240922124951.1946072-3-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E66:EE_|DM4PR12MB8452:EE_ X-MS-Office365-Filtering-Correlation-Id: 5ad602e5-8376-4bd2-a7e3-08dcdb0523be X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: SGuQKOkJWk0IR9E3xg2q7oiN3LF8iuk0fk6AesU2nZxXKzLlz86cdFH9Uy+1XlP3ohXtyCyTXq9DOvoWL/qdYAw53k+UvQazvO+pyhE7uBowTSDpDNdcQP6cfk7LveoEab1syX8v8SZNSLTtTbZkycZ+RtZe3UJEHJWUoaLtXk4pVPkahYp8Py64olorlkNEJwfLbLZNsdwLY0rN17/B/o6rc2aMnCMmJ+IdMLSt6+j6fMWwfx6kSSDeOIaLMvXpJe3+A0HuCgofiuhWJ0SAeRbLoimhWxCdu454JZAlJD97hJb9VJY35tiFHB6OjfkuRnf0kpXcXKljTzS519ANqPMAXUY6vpCHTh1C7wGJPl01PoenInboxAoSYo2RrjeBgCAWQYJFAdKiIE1qDJP3G/T4Ah/bifJXg7VYjSGZ3sF0vBX6Fl0Ig8c7ujCx5wfXdzg4BvmTNtjN610gq9oCyjkp9xpjcq1Oxbw4Q4cQ1cIjyGUn+0TzDfmapE1WovRttn56JQE8rHtNoMrwaX1S6zTmW3+RfHpHwzQjU2NZhJ22UM+/emztBnftyaUfLINzGuD7XsYE4lwZDcAJA4zENea5KuTA/bjxCFXOec70IZzxq9XgryX1l4JvMV/Kb2kcvvOZPvvi6eHPCpU+len5K9k7hpcKcXvhTJTS1V3PUO+KRdB0xj51OyhuSSZQmAOw5OeO2Qxg/Ena2/F2RLrRzm0KKI4B3P4mmX0zb0sshOsrDONLq/Vz7NsqCydDeMh90VGukmSEB2IOuVFsULWa8COMqETjknvv4Jh77SVPTYr6uxrOIsKCzKhpDzYwholCst79os105uY6BiMidhAhTLp/tjzqY1NmpWcg2itjA4Wxt8PrmJKgbL+h26SIx98yl3T2OT8yIVEDD/Vm4Li+0OjGNdjUmrBHW0sGjrtAeTXSWBf2s7Pc6X1MWpc/cdzoss/918noNYiS6Fq0+pcf0g1RMprDhIWS0FXf2BPuFV0JmISwEJEiv71+BO5Gz451aTgovVEvkxJuuXUEuOntp5uJ2Gh9eEpxNr5qKGG4ZsemLHyGfaCFv0ZOlGFCGzuMOghkEx+llgH07HcHEZLp1r5bFYOl2Q54Fj145k7mC+iH7nlkv4dCl2HXLNaqIil3DPJKIDe7Ux+og//MDzpavdvDlnrnz4nUoUHScxKUXreGdVWyh732nVw7wK9cBE8cFSyG+RE/9OPHvSzyMjYaKNe5yVzARJ2tV0fC3lY+f8zLmLGF8CI+Dm9DCgcxZ+hAhKFtPxrbr6g1mJYyBJjgeYk9L7PxrVXJ4//QJyhV9BrvSvrUHJUqOfDWK48zGDi7dFjcM917pUOBqUJ8OU02v5D51aVwXPjnJalTgjo4LTA= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:29.8753 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5ad602e5-8376-4bd2-a7e3-08dcdb0523be X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E66.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB8452 nvkm is a HW abstraction layer(HAL) that initializes the HW and allows its clients to manipulate the GPU functions regardless of the generations of GPU HW. On the top layer, it provides generic APIs for a client to connect to NVKM, enumerate the GPU functions, and manipulate the GPU HW. To reach nvkm, the client needs to connect to NVKM layer by layer: driver layer, client layer, and eventually, the device layer, which provides all the access routines to GPU functions. After a client attaches to NVKM, it initializes the HW and is able to serve the clients. Attach to nvkm as a nvkm client. Cc: Neo Jia Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 8 ++++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 48 ++++++++++++++++++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index 3163fff1085b..9e10e18306b0 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -7,6 +7,14 @@ struct nvkm_vgpu_mgr { bool enabled; struct nvkm_device *nvkm_dev; + + const struct nvif_driver *driver; + + const struct nvif_client_impl *cli_impl; + struct nvif_client_priv *cli_priv; + + const struct nvif_device_impl *dev_impl; + struct nvif_device_priv *dev_priv; }; bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index a506414e5ba2..0639596f8a96 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -1,5 +1,7 @@ /* SPDX-License-Identifier: MIT */ #include +#include +#include #include #include @@ -42,6 +44,44 @@ bool nvkm_vgpu_mgr_is_enabled(struct nvkm_device *device) return device->vgpu_mgr.enabled; } +static void detach_nvkm(struct nvkm_vgpu_mgr *vgpu_mgr) +{ + if (vgpu_mgr->dev_impl) { + vgpu_mgr->dev_impl->del(vgpu_mgr->dev_priv); + vgpu_mgr->dev_impl = NULL; + } + + if (vgpu_mgr->cli_impl) { + vgpu_mgr->cli_impl->del(vgpu_mgr->cli_priv); + vgpu_mgr->cli_impl = NULL; + } +} + +static int attach_nvkm(struct nvkm_vgpu_mgr *vgpu_mgr) +{ + struct nvkm_device *device = vgpu_mgr->nvkm_dev; + int ret; + + ret = nvkm_driver_ctor(device, &vgpu_mgr->driver, + &vgpu_mgr->cli_impl, &vgpu_mgr->cli_priv); + if (ret) + return ret; + + ret = vgpu_mgr->cli_impl->device.new(vgpu_mgr->cli_priv, + &vgpu_mgr->dev_impl, + &vgpu_mgr->dev_priv); + if (ret) + goto fail_device_new; + + return 0; + +fail_device_new: + vgpu_mgr->cli_impl->del(vgpu_mgr->cli_priv); + vgpu_mgr->cli_impl = NULL; + + return ret; +} + /** * nvkm_vgpu_mgr_init - Initialize the vGPU manager support * @device: the nvkm_device pointer @@ -51,13 +91,18 @@ bool nvkm_vgpu_mgr_is_enabled(struct nvkm_device *device) int nvkm_vgpu_mgr_init(struct nvkm_device *device) { struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + int ret; if (!nvkm_vgpu_mgr_is_supported(device)) return -ENODEV; vgpu_mgr->nvkm_dev = device; - vgpu_mgr->enabled = true; + ret = attach_nvkm(vgpu_mgr); + if (ret) + return ret; + + vgpu_mgr->enabled = true; pci_info(nvkm_to_pdev(device), "NVIDIA vGPU mananger support is enabled.\n"); @@ -72,5 +117,6 @@ void nvkm_vgpu_mgr_fini(struct nvkm_device *device) { struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + detach_nvkm(vgpu_mgr); vgpu_mgr->enabled = false; } From patchwork Sun Sep 22 12:49:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809094 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2073.outbound.protection.outlook.com [40.107.100.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 842FEB664 for ; Sun, 22 Sep 2024 12:50:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.73 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; cv=fail; b=bpLnWeipQu0jz+xM/s7Epd+ugtll7q4V7m6Mztg2y6PW9Dn6BbmUEl+FpW3OmuJb8zFh1MX27vd9T70k1RC2AwAyRyd2+jRggSZMXmE/6kjAD6dv5KbO1tgyFBeGi9iO0VlrxrXy9KQXLUQLdmtPnO9qUuaZuGYerdnlKY7FBE8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; c=relaxed/simple; bh=wL8w7RiGax4Gt4YhGvlxPcRxkbv8PWul1HK2fg9jxW0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dNCQsKZ9pdK0Q2gursGHnAMy2dCGz1iSt6O9dRGacbq7ep3zHeapcogKngrvtqNVyyTr3Q1eFbANO/22dxBX/6E+L9/HtnHj4Ha76KZlb3j/VcV56BykAX3ftH9tMJeRlpd9A7+Ltz4Em8WbyovAcVrj/SHGizKKt0lQ+u+GilA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=bsb+eXqk; arc=fail smtp.client-ip=40.107.100.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="bsb+eXqk" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DcNzaUwCK7h9FjGmWhoNusSQcN57ACH2snh5qpcoieiSSmGMB74kDD83HpE0cDmYo6g8J8NJCqnuiI3F1OHukB42jcL/Y6kPPB5ggOdZXEcuaXU00EmHtp21XfQdHwbDI2TnoOACiuu4qs6P8ADYeQCsByesMyV+d2OOWtxRfDvsrN/TvqlxpvFvNqS6lKU+lvJrGqxtn6cjEtzW6jtzbknY5mzFhD5b+X9hPPZeS4+CnxaOrPLHAGgODPFq8CNRdD6p1Qcrr8yYEXqjHXpKoUET32vl2FbYOvjyrFD/oS09jHB0AGmJtnNdEegzo1W6kbYTSs+ethl88OLqoVbE1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9HtPBKZcitq34yHYeW8ZKpZIUkfMnrrplcQ0Ok99de0=; b=PAuT3DV3zYpeKRx2qvnL43RtG0kLkTPNis7VeSET6YPrklSxDJ0y48e/qVcZjiK8Z7JBjRliu4JRjOapwl7N33TkJC5/A6oBQRmdAe3sBU+r1Mb0KX70MIK3y5nJk4tpWmmnSCNf0NwQMg5Ea0Lm/seufao1cDpZCognflJj32NWSqxYWWEm1Yg+ijO3GzUhaY024jN5kBWvynaiQM4+1Qu83BS6GOmEX+ZK/qQ0/+h+qKWgrUsblHKVdVyBNNXBok5pqSxOx/yLiB3g1MSU7Rt2fX+OO17jBY2QFGRQTUskET8ZDwQ1iN9h8ORtD8B4aC/X4EAm8Lw+vMN5FffxUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9HtPBKZcitq34yHYeW8ZKpZIUkfMnrrplcQ0Ok99de0=; b=bsb+eXqksUXz3ICIzKj3w51FcOtk2xxD2RQ0tBjebH3qsuuu7ybstfjSGUe91HoI8yHn+lgJsMQsQN8Cn1tMcUBfDJ3Zo3QIcAVjenORBsmA76Csjprqfy6lps7aEY2p87NFlsTqSJreoHgbgvRcoaWRhVImdhhkDLrzoM/UC7suJ4tv6zl4p7ztlaEpzGvKdHU/AaBFN3oATNpyxE1PLJr3eRbXPeuIu4KeL2HBaAQgTFIAyQ2EA2rd6wNgDBp0KKdP3W/Mg21dt6z/KITemNjI57UQ5S8pyLZRGa8iIHdhUqgWZqaqBL2rF7kgh/VuT1fQhzFxClZZeFRHP6M0Vw== Received: from BN0PR04CA0148.namprd04.prod.outlook.com (2603:10b6:408:ed::33) by MW4PR12MB6731.namprd12.prod.outlook.com (2603:10b6:303:1eb::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:33 +0000 Received: from BL6PEPF00020E62.namprd04.prod.outlook.com (2603:10b6:408:ed:cafe::3f) by BN0PR04CA0148.outlook.office365.com (2603:10b6:408:ed::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E62.mail.protection.outlook.com (10.167.249.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:32 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:21 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:20 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:20 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 03/29] nvkm/vgpu: reserve a larger GSP heap when NVIDIA vGPU is enabled Date: Sun, 22 Sep 2024 05:49:25 -0700 Message-ID: <20240922124951.1946072-4-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E62:EE_|MW4PR12MB6731:EE_ X-MS-Office365-Filtering-Correlation-Id: 11a7de02-27fb-439b-61c5-08dcdb052595 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: SH8NjsTALsyQLLVXYoIDym+AGnuD5Gge7W37SntOokejbaZ3NpxhTX0oU0XWYN47tWjK8lltsnEr1bYV2F52G7oDCcOWXGQEOleMZUBUXu5DQ+hf9JmGBlZBdf4QSyCZB5JnGtk3RCU6ebyMFfVZtnn8ejVLJQUQ1w5WixX5vAA0LBzAOCuKzwbZlQj2fEq7kcno3W7sdds7ovvoOAk+vyWIeKdXRPocGogZRwgslSsioMw5gW+ds3WofkjoED2EdYbf8CX1KXhS083tAqGRBYGcYuljHRCMLcvfLKldfb78vM2bl7hekJyPKeQ/luSc8A/3PlUAiojX06afMEXqBNThec3YtzYNecB8XwGkZTClHh+5o03cCSHfXxxA8/G1mEErmxV3zjh4VpJUkkaWgp6IWGhbQ5Ku1aSGschliBqRmImK8rWVhwK+DooAcoOigOqSFyuokpna201OdIHzeOZ9x1pDJuXVKKVrmM/9fkK4A1edAeRw0Oys7T3ROnXtN+Lpr6pd5ZyU2Fv8BovZtj6Y7/oK5kgIG1bVKioZkIiibro1Ds8HeoCEpDjp8nDLDp+cqj+pMJBmbvlZIdv7Vmxl12BlKLWf6rULaczZoMX7FAEFmtOayHOcmygbQ4fa4XFRcsJkdlXgrVD+RYWl1l9MemTSiW8ej0z338cFtPh1SPg5sve7z7hvGDoz5XG3fJaAvIoeHZN1QSurAMEgZndr0bB86Ms3XjS3p7s+vMVXCxrHGbwbTgNO4QizxO11768DIemqBPV+Ijrzn6Fpcbi+qXjwZS4oLqgA0SNuAbeRgn2Cw9caEWBwI4u6uNVXkUbNVKfwYZzkxfcGt2R119Ul0PU1FKPj/0hQsSgm9z3997T6uOSVBvT5TAKS1y45JPaMw05C8xSRT3qt6oXJ0sOPNv+5I0OqunZ9eUa3M5Cx1f01EMMXRy+ZW2Ce4xfS609IXVtcmG8Lr4G+MBvw4LBqhdaiw/5VY0NJwrHpJ8/pKRjlamdyJCwYiAa/nxhrODFBASUzEDSoGNKzzYpJnIGcWtlYrwcYAaBpXbIvtG82v/DfXrsKqVmnHpFAu+rIbwytWXKvA4cCY1jNxCv9ml44mX5wA4yFxySufu859X6mVsVWihWiRy2T0acnEsUC2O47qQEFEI0xJFv9KKg4s1EyLpkuaMbv0yNb/qk6YH77/Ipj5Gd1tTcvmEDHeYDFxlPM3Lcu8Qpm7LRGkJNdEIgwN2NtyUjdW5bW5UWAsRiHMOc9blUzfTGpB7ofdmSKUFDp1zFdEMfHZ2euQ8QdBaCnTUe+xwar0XefSFKMb1Kw26Qyv6hSODbYFPA9qfucGRZvF00CPniMqZnBZHoIYQhQjFVx54Y5J/I0JinDz/Ym6jZvGBiYngiqghGmpAlLncQoTt8NT0SJljCKOhPmoXQieaS3Md9Zu3Ak4uzpYXEjKvEBSWejGVnYnGjiW2+B X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:32.9602 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 11a7de02-27fb-439b-61c5-08dcdb052595 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E62.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6731 A larger GSP heap is required when enabling NVIDIA vGPU. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index 18bae367b194..a38a6abcac6f 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -2638,7 +2638,10 @@ r535_gsp_oneinit(struct nvkm_gsp *gsp) gsp->fb.wpr2.elf.size = gsp->fw.len; gsp->fb.wpr2.elf.addr = ALIGN_DOWN(gsp->fb.wpr2.boot.addr - gsp->fb.wpr2.elf.size, 0x10000); - { + if (nvkm_vgpu_mgr_is_supported(device)) { + gsp->fb.wpr2.heap.size = SZ_256M; + } else { + u32 fb_size_gb = DIV_ROUND_UP_ULL(gsp->fb.size, 1 << 30); gsp->fb.wpr2.heap.size = From patchwork Sun Sep 22 12:49:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809090 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2050.outbound.protection.outlook.com [40.107.236.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6568E2CA7 for ; Sun, 22 Sep 2024 12:50:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.50 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009437; cv=fail; b=bxHODtVmguPP+GL46m/KN5QSm9lISefRly7rVBtc8jNfv7g7rYsG4BtcLFoyByh8YYuwEe2LBB1kYBjchQwNLaB3uHwBpJ5DcftKUD7O1uUdGOHu++VjpBwB5tbTIDjrGnl7jHvmhfsb2uROwllkMaKRs/QOar+CowKoQuaKkfI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009437; c=relaxed/simple; bh=sCNz+ZGh+rCMZ9tifjvFxo+/23kwlYpW7wfCu9LqW/k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=IgfCr0ejp1OAmYjIfCkkv+xrH51lDUbOErGs1G5DVJ5GmQDJxEyiCJwZFfyVF6VF9Jxox4DF11K6j51IuKbxJ1MI+KJ2D7T+GnYqvWX3hdgUi/6VQ10ukyG/UAmGWxOVBxHlnOkwMzhmcMRoyBReIlGgVcrrZYbfmjqO9SGcJhw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=gg+5YNgx; arc=fail smtp.client-ip=40.107.236.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="gg+5YNgx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Q5iX5T6hm2BeOuEyM9qfPaiHBLbPB7Jv0Lq2J47qcLMqyLNZprzB6l08WmxTZQKzhJ2sB0XMyge17JggGG6egO2QKFw0EznfXJtZ0t+zOYQrNfRCWS9K+uV7bmB2oBQoEbFbjtA29vmnb4oEtDeR7J4WbtKJn5YzYzUkC+u5OIcIIsxJj7sSgIfLjq705iWB1q3bWpNQVnu6D0mJYoVAJ9l6Hf5rrGkmxGPZ2FWegk6M5ni5pt0bxIh9GBK3Yd3f/RmteyhEWWCq/RmmAT8oR5PMrycFYFGEriKL2nNqNl3T7weDNfR8P6/u/FMF4BZdSsqW64Cpb+DqoVPcwgt0ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B76wO9woObFjAo5fjIDfWpTEn9+rDOcfspKJJKsnWKA=; b=E9ARcp4MPyswUZnnCxEhN8sdnBbE8qb0FYE66oLUrznjGbkU3AkEIZa5yr99NcgYgYhvfqk0bFzbsR15BKPucf99RtHPb81Kv5bWp9FXcgGwcyx2MkuzR+AktxHKX5jA0L8mWibS5xJxn9tsRYoUm/CNHk8/jLpspugD0y7qzbEimtzUSi1hQGzHBEOa1FJ0CrsoDhnj/Jz0b5QUCG+uX31gPyha2+ESpyUPT19ntH56CGCgxTdPWanA6x7MHNDLaSu1sX/qPOjBUCMHj70Wq+7EDx4WAhbIwEjheyWXVpmIlvP9U7d1TjCQoU9OesKo+8mfuyw5c7ZVzU/O4M5hFg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B76wO9woObFjAo5fjIDfWpTEn9+rDOcfspKJJKsnWKA=; b=gg+5YNgxSXFO9U7uMZy6blJ7YudvbcMSVREUbB0Yt5GEZDO3fxBavzA5T9TZAYUMbzzlfAWTWboBojTaUWIngUg9Rw1xnGJx5Z0ee6q9s6RD2RBB7zeupbvo+b3EIcZkvMTTKPtSOanetpiA8Uav1ZJYwdc/oWnyK32UKqUFB1qxhGUiKKYIg/yjFaML+DI1jn2r48+J7jZyiRz+lh1h8aTdD0oPTcAJJk0omsnyOSbGX83NyBeSbjjohKQVdmCHq+L/WnmOp3cwqvnJl8f64YhL/hI2bfWIUZGZhPsCU3kFOOrXbJaHyV0YusdOXNp2v14QvhfhKCYRUPaP9fXbvw== Received: from BYAPR02CA0044.namprd02.prod.outlook.com (2603:10b6:a03:54::21) by IA0PR12MB7751.namprd12.prod.outlook.com (2603:10b6:208:430::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:31 +0000 Received: from CO1PEPF000075F4.namprd03.prod.outlook.com (2603:10b6:a03:54:cafe::b6) by BYAPR02CA0044.outlook.office365.com (2603:10b6:a03:54::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F4.mail.protection.outlook.com (10.167.249.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:30 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:22 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:21 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:21 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 04/29] nvkm/vgpu: set the VF partition count when NVIDIA vGPU is enabled Date: Sun, 22 Sep 2024 05:49:26 -0700 Message-ID: <20240922124951.1946072-5-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F4:EE_|IA0PR12MB7751:EE_ X-MS-Office365-Filtering-Correlation-Id: 218d2e56-2df3-4ebb-2bc4-08dcdb05241e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: J99FfgnkxePME4EdxBmarsA2VFSUad9Nd9L6FuIuZbdNt9wcQ6AbAan5NWoSA843M2MPALx8UiwH0VHjM3su80ObYsSYyED9X4xxQqrIxadTfWD/ED9gHQi3197k+H/Cklf8UEXnCf4zQxm4ZY+7xZwIzWWwEcximMCS994YJ1VQvPea7CaSGKwQDitH6lvkgQ+9ExLwb/UlUXDv5nPDl5F5B5crCfrIg7r4se37/16CyfQdx5iFpYxkVhrW+xyO3fqYFud+7BbKzFij+3jaH7+Mz/Jnj0eQlZfO1XKfK5AmrtQFLLRQ67g/LU6RPP05b0CVRAZyQ0Aoq/BD+tGLCH2FW5Ywzdi4xl2o42vg811F2rNHyENBMNA0s07vCRmJNZFAy+W+d0KYUTeX34F3q0jw94Va09/2jSXrDj4KqeWkg0i+9RM2YnrFeLVFVpXEsaCrBLwscP21QfMuNWag7fAJY8+ri4xBhvC14XjX3tW8CzBQCJhUM5KypXnk+koqQLm5AOMD/Yb/UJtTzPRRt8Cp4MUKhenIm5HYEUqiVFkvFWMa2wcPSct+x+mvQ/hsNVc5oEh/WOWa12XasYvZ55apkBGEtQaj8qVF+dgpazcUeUyvInQ6Ne8UJDFs0Qcenw2FK0VqSDT+dsnaKyCD5+eooivjouNM2W31g4qRagwsDfZrhpF4p4mRkfHzQI/QqymHtL1QbETsb/OfFi8RfUZPv90zFRTQvHFQA/aiTqJdlBxU9ZTcsDbTcsge0aX/XzHvmKVeJaIlxNKUi+6i4mo6D2DJhOoSFKk8H6ityoujNmIhcBhnKTDG7UtywRcnMWOGlATFTBxVC8o6A30qV4OluSUfsFJgGeA0Tx3dzlLaPyc2/FTbMKnzOPQUNCehKd8hOg7AGsyjRUTQUvJIpoZF+HYdAcikhr4fGpNSUV4WSJ3m5zwxIyrTz/Z2Hgm4YH0HLAm2MWtEAqc5m1GwvQHLVuJQ1CNcvKv3V54UMRXeo/s7tt8CGLN7vlE/u0RI7BeWli1DZA99CoJYbQZIwXljMqIXT1s9v1uy47BbXFwazC1vOUl66aWGQQozU8zNYLCwq/J/UkL6ElhgGpBCT9JJVaZeIlP2X3w5gAJXyvSgvyw55U3pAKtn1jnEXGhRdPlG/KoovUYkDgLVmaifboU0PndJgGLR2NDeHcfT8Mc4WoCBSwDxDuexh+Dn+G+5UUkTydvdvZHR360BGgMXf1ObMag8QLOhsppu39xpPwYOhQrqoHqeSgd/5E2d4ODSikN9E0Pyd4dXc3OUxyUnPEyDtgC6G+Q2JqA0FiVFjycWPail4R/TMMFk1Z+f530/QxDHAHxeCBfn7ef4sMxBg/6Quw27b/nXs8jkd03NWk/N79rNJjdLQO6dpZIMy/Gr X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:30.6263 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 218d2e56-2df3-4ebb-2bc4-08dcdb05241e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7751 GSP firmware needs to know the number of max-supported vGPUs when initialization. The field of VF partition count in the GSP WPR2 is required to be set according to the number of max-supported vGPUs. Set the VF partition count in the GSP WPR2 when NVKM is loading the GSP firmware and initializes the GSP WPR2, if vGPU is enabled. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h | 1 + drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h b/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h index 3fbc57b16a05..f52143df45c1 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/subdev/gsp.h @@ -61,6 +61,7 @@ struct nvkm_gsp { } frts, boot, elf, heap; u64 addr; u64 size; + u8 vf_partition_count; } wpr2; struct { u64 addr; diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index a38a6abcac6f..14fc152d6859 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -2037,6 +2037,7 @@ r535_gsp_wpr_meta_init(struct nvkm_gsp *gsp) meta->vgaWorkspaceOffset = gsp->fb.bios.vga_workspace.addr; meta->vgaWorkspaceSize = gsp->fb.bios.vga_workspace.size; meta->bootCount = 0; + meta->gspFwHeapVfPartitionCount = gsp->fb.wpr2.vf_partition_count; meta->partitionRpcAddr = 0; meta->partitionRpcRequestOffset = 0; meta->partitionRpcReplyOffset = 0; @@ -2640,6 +2641,7 @@ r535_gsp_oneinit(struct nvkm_gsp *gsp) if (nvkm_vgpu_mgr_is_supported(device)) { gsp->fb.wpr2.heap.size = SZ_256M; + gsp->fb.wpr2.vf_partition_count = NVIDIA_MAX_VGPUS; } else { u32 fb_size_gb = DIV_ROUND_UP_ULL(gsp->fb.size, 1 << 30); From patchwork Sun Sep 22 12:49:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809092 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2051.outbound.protection.outlook.com [40.107.212.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D16F5321D for ; Sun, 22 Sep 2024 12:50:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; cv=fail; b=NTuPmfcu7lYKaO/bvynayrpUCEn4oyxPqyduuGyjWJ3HB7XuAnPGnfSxFCHkJ0/qQ3IUruE0fak1bfheq/qu/zw8R/j7C5IBIOVRU+H5pnCAhhTCnhF0WuDtCqOCRRXdbUyqEj3bEKbvcDMHoG5QnskEVvZOUgIYtwjNYXzxfc4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; c=relaxed/simple; bh=zwOqDFNxAwmPTFe/jiPwyXD4ZPFK3OZYQtggKJAz0Uk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=G/15wBY0Nq/modNM0EwIi62uZtUJRYjt5C6U9+VnZa4TAVaxC5zH1BozS/woORP5ez3PR7T3/nGZ+hDIHNA6Eh4Cftd16pyxIeYNH0nZIRbkL6U8x/+YPpZklfeQnh1IvIcetgVE7RT9l8zARiQ6Ds5nyi+ziX/SYSseoin6B3M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=B80vG2Rn; arc=fail smtp.client-ip=40.107.212.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="B80vG2Rn" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OJahA80M8O5Ht0RPvbY214DUowsPmko39BfxSbBwJDF584BPiKezryEUB45z8EC227gvoJkosayoily2SUacgOzj87dUD7aCeuH61PlTrtOtIrlv3du+MlQxwQ95SVQ6QqJmj/vq9GuOnixj0TzUbhX9/FsOG2uE1jPTEDsUuNnihQowac0n/ZKTXpczJoumINH+DB/FZaVGGmi7/R5fkf3axicBbBkjIkCzH+xJg7qYWdMPqKH80ZB/LbvbOHUStRF77CANzOzl2peZsDcjht5G4I9wYxSdmQyZjcU2nbeO9i8erpVsyjehzm2eWgffGiIOPUNOx80kcxcestlY3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=t5Se6NOULlWwT9b5/k8yOVdePxylOx55yZeC547knhY=; b=xozk9X0wiHBsAqriDYDxYLQbhDoqhYZr+S7j23S6W0r1epbgy2SuzNGBbfm2d2uddgzj9+zOLYELLExYYM2YBI3XmoKHacaCamLuuzgdMdrNm4WnZtrp0nMqA1f5b9gUMmMY/imdBPw3YF4KifgilOqPAN09XhpRm3dJdl/gL1/xaPz9AiRXxq8y5dwZ34M0e+6haNXjrqlTuAofH7qIfK3yP8GPp+TIovWCvLrNfmioZ6k4YeLIbsIaYRZx5roqvaS2t+NFvxPdiaM9V79nsCHGGSYk/Wvdkk8BUs7hI4dnGGdDt6VoqNlNFEv0y6kA++5RbZcNZ28gjj85DSvSiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t5Se6NOULlWwT9b5/k8yOVdePxylOx55yZeC547knhY=; b=B80vG2Rn4n5mkRtdrCV5g7X840e5mSo5LeoBtcr+MIldwldBAGiOoW2aHqEB+HuxOq64bd/zHR5u7d329ZBEHE5qGVbyzNRYXuLx9kuYrfydj2+mgiN2hHC3yc0f6pFGCI4dVxWK8Zo8tYLgveM6jsVMS7B36MiJTrM7wVuD74TvKjZ8mJdzpWYvjyiIjHqW2ddQxLFyZZCH0BfqDJoo+SUx86NXSGkA5TBZvwCi/KChM+hTcFY1TkruaX/V//2gPmCWfX6sjJJ/Wpzew0DDs+JmM2Or/5l8tT0qIppgFAugApoekUo4zyZFSslmIX131X26Mq+E8eIbATaTmF9i5Q== Received: from BYAPR02CA0039.namprd02.prod.outlook.com (2603:10b6:a03:54::16) by IA0PR12MB7508.namprd12.prod.outlook.com (2603:10b6:208:440::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:32 +0000 Received: from CO1PEPF000075F4.namprd03.prod.outlook.com (2603:10b6:a03:54:cafe::d9) by BYAPR02CA0039.outlook.office365.com (2603:10b6:a03:54::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F4.mail.protection.outlook.com (10.167.249.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:31 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:22 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:22 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:22 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 05/29] nvkm/vgpu: populate GSP_VF_INFO when NVIDIA vGPU is enabled Date: Sun, 22 Sep 2024 05:49:27 -0700 Message-ID: <20240922124951.1946072-6-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F4:EE_|IA0PR12MB7508:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ce16585-104c-4c6a-fed3-08dcdb052492 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: PUxBJFbJeOf9rK7theTz1g/fZmxWoAuYGMKHibxG1BH5C/jFl6EwTeHNQHgnpq02KT3JlNoIgPdWpgxU6w7y5mZYYtFc1BnN2boP849e2jbhj8AuMOU2HUSQtUDlYj1I7J1sMB7rUT9dYs+W+MUkZMe7Qto/xZvdfKSm8wJU8cU40dApUpKZjD1yxBtCtse1oqnLaSc0uB6N0xD7fMDS3kGLaxddKdblpb1/pBn3f8Gi3dsUm/6RPavkmN4cOqd5A88WLjEBMtww1+14og4lMshGkmQk36nomYVi0n6xnJcQoLbi+xnt6r6egosOWAQXkq53dcklOBRmJKVXYzzXTrGMipdHidNwKY0ZsL428neBu6/dUiE+onMl6UeRFo8rcRiijzMzEdpxLqkFBY3yAZVaN1StPVBF4ohKOD7rapRcD4pscFuUqQqCUu8stflVr6VRst3ATHXscD1OwjOFlrIjxUexCWc7rynionZXtZe8c5RTSTGYnjMbWhgnfLVenLyGLxhvoBnJI7PV6ZOMAsWFHB4pPIo2ex5n51EPNHDVxG6P4q9wyUA9bikjkbLDS+WStoaCkiZQ2i5QF+M3UD1zzHcSpr1d3Ek2akb7ifrnk9+bYOg8W0GufFRspdsTEpYGxvloaZ7BNp+WO9ic+7LChy0S2VU7rK9PSmhDgGPeDVqVtEoVMqCuboNdcb1dTZGngfnWzmD4eprzOdFPt2oBH71IrZNgpjKW3fzmGKohCwOeZMVG3j3m6LRFbLtdCjuyelMkOAZWgM46CWnpFqakT3N1EqjfFnzJLuUpPMQdVhjzVkHyk+nCY8+FJ9oPXKL+jrE+r+3ILaCl4M/oM+vhMDW4udu7YGRoHAb2Mbk8FKUzmwApMCxOurISj3ffel9mpRO8+HqgeJJKFfznBK1QDxUbRWO7IZBa2kyfJlwruvZ1PQdI4A2+ebdQNUQxDdr1HVtsbMjEGcTwrrr0svMrniWfWZGWMehLi8Y6/QTVTNlO0K39D/Nmxk2t0BiOB5d90pYeGqXmP28iZlwOAoYAo5rFNL0oFfXuMaoexFDNNq/xWE4ASSpdiP4EHGRK9YNEp9gH5r6ypZFvAXeNxlClUSOvq7a0zaOSsoHVqYhF5WvbpAC3+ZAA1RU46kwxkY/Fq3E4bVoJBD8V3XaNfboUoZ1jrQGCCArVOB8TjW7geMgDZk+7tHW7IJdeJRiPZNbIUUjxygz7SCa0GPtDFis8H+jdCr6AZGomVcy+JBBcvx1TfxnA3fD24xTQ06XIGC9Ski8USBZXkLokhzBSgW1vaAti2f/hBZ/DMVvxofF42CyBKLQeHao/bXxr4Tu7opqgujx4uxLIwLw0TusmgMgiSwEIYS5V9XpRe+Wrj6Q= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:31.3762 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ce16585-104c-4c6a-fed3-08dcdb052492 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7508 GSP firmware needs to know the VF BAR offsets to correctly calculate the VF events. The VF BAR information is stored in GSP_VF_INFO, which needs to be initialized and uploaded together with the GSP_SYSTEM_INFO. Populate GSP_VF_INFO when nvkm uploads the GSP_SYSTEM_INFO if NVIDIA vGPU is enabled. Cc: Surath Mitra Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 2 + .../gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 3 ++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 50 +++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index 9e10e18306b0..6bc10fa40cde 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -21,5 +21,7 @@ bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); bool nvkm_vgpu_mgr_is_enabled(struct nvkm_device *device); int nvkm_vgpu_mgr_init(struct nvkm_device *device); void nvkm_vgpu_mgr_fini(struct nvkm_device *device); +void nvkm_vgpu_mgr_populate_gsp_vf_info(struct nvkm_device *device, + void *info); #endif diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index 14fc152d6859..49552d7df88f 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -1717,6 +1717,9 @@ r535_gsp_rpc_set_system_info(struct nvkm_gsp *gsp) info->pciConfigMirrorSize = 0x001000; r535_gsp_acpi_info(gsp, &info->acpiMethodData); + if (nvkm_vgpu_mgr_is_supported(device)) + nvkm_vgpu_mgr_populate_gsp_vf_info(device, info); + return nvkm_gsp_rpc_wr(gsp, info, false); } diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index 0639596f8a96..d6ddb1f02275 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -3,6 +3,10 @@ #include #include #include + +#include +#include + #include static bool support_vgpu_mgr = false; @@ -120,3 +124,49 @@ void nvkm_vgpu_mgr_fini(struct nvkm_device *device) detach_nvkm(vgpu_mgr); vgpu_mgr->enabled = false; } + +/** + * nvkm_vgpu_mgr_populate_vf_info - populate GSP_VF_INFO when vGPU + * is enabled + * @device: the nvkm_device pointer + * @info: GSP_VF_INFO data structure + */ +void nvkm_vgpu_mgr_populate_gsp_vf_info(struct nvkm_device *device, + void *info) +{ + struct pci_dev *pdev = nvkm_to_pdev(device); + GspSystemInfo *gsp_info = info; + GSP_VF_INFO *vf_info = &gsp_info->gspVFInfo; + u32 lo, hi; + u16 v; + int pos; + + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_SRIOV); + + pci_read_config_word(pdev, pos + PCI_SRIOV_TOTAL_VF, &v); + vf_info->totalVFs = v; + + pci_read_config_word(pdev, pos + PCI_SRIOV_VF_OFFSET, &v); + vf_info->firstVFOffset = v; + + pci_read_config_dword(pdev, pos + PCI_SRIOV_BAR, &lo); + vf_info->FirstVFBar0Address = lo & 0xFFFFFFF0; + + pci_read_config_dword(pdev, pos + PCI_SRIOV_BAR + 4, &lo); + pci_read_config_dword(pdev, pos + PCI_SRIOV_BAR + 8, &hi); + + vf_info->FirstVFBar1Address = (((u64)hi) << 32) + (lo & 0xFFFFFFF0); + + pci_read_config_dword(pdev, pos + PCI_SRIOV_BAR + 12, &lo); + pci_read_config_dword(pdev, pos + PCI_SRIOV_BAR + 16, &hi); + + vf_info->FirstVFBar2Address = (((u64)hi) << 32) + (lo & 0xFFFFFFF0); + +#define IS_BAR_64(i) (((i) & 0x00000006) == 0x00000004) + + v = nvkm_rd32(device, 0x88000 + 0xbf4); + vf_info->b64bitBar1 = IS_BAR_64(v); + + v = nvkm_rd32(device, 0x88000 + 0xbfc); + vf_info->b64bitBar2 = IS_BAR_64(v); +} From patchwork Sun Sep 22 12:49:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809096 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2049.outbound.protection.outlook.com [40.107.94.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF70BBA38 for ; Sun, 22 Sep 2024 12:50:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; cv=fail; b=V8Q9nlw1q1la+sw+/mTDOD8N0AY3NggEsz0McYQg/hSAIOKW51Sozr5V2YSsoExNU7FhGXFqtvjAVtgmMpqBiCVIaaI2NfSFjhz9bWiSYBE/fyXryKZF0vw3XuNuSW3ZXAHHAsYwpuWWNR0wF8KQrdbNswnndTynPTnlGgsJVXI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; c=relaxed/simple; bh=6QvQS+dGPwC+NX5O8iwmPDS1q2pt3ur9kRWyTE5utOE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gbYntLtiHmVGlToBV3txKrg0JneEOY1za3DaAxVWUdmLCPqH0jsBS02+r3dZ5fMgzRLMPZaZ/SwPH6GPMRbWs9Y5bDpQPUy4XbLIflZ17bWP3s1GCa2HGGQbu7J8omrn98hMGKT2pmiDY9yDpEwUngQA+1nPvb1s0KXWwtN7VSA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=jKHfch9x; arc=fail smtp.client-ip=40.107.94.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="jKHfch9x" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uNMdM773AKB2Hh5H4FQ2Y0Kb7FxAQg0964rTTArQPWxs6Yd3K80xY8Wgq41mFd0sufyHm0qE36zZZr/G71aQw39+6Gh1vdxYsnDDIWaltFmQtitGOz+BNFNlAmU1smPJHO3J56xHmkwCVCjgKsc2cspsyhcAD/CMPfrDkRWGqtR3iS9SOn2NlRN+ZGEU2AZTIPcecWMwN2ulGTzxcOGMOEo5gbOkosUeaZg0O+XLH77ZvH8yYwBZEsQIVs5ThHIK8dEiY78UpOGU5uyuUtSaxLpxpFxAj3UY+wtgjDkIy9V51d8tYmaSn/5rPMb9Sy5PuWu1xhTduehWgSSp1OcggA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YZusEDF1qzJX69Q76nZVilAAz+yoqKvuL8KWAZlvxXk=; b=CGhOcpLeoibFX+HTHd0kxKmKETc36S+fNzFGpsZ3Rcgt/tgtCCV8M+jD1wR4uLYTBTA9rKP3tsVW6lmcJqCb2q9ogetKctylG0piABE/EuC4Wj1m/bLfhSu0z7BTrfG2t+G8/FbHkGc/GiP/gJc59qwcDWrb7Pkaxk7BIKxBxVNSw1eXIuABz1R/XqE9KBz3Ow/A866dPm5p+0SDRyOXP7kI9tGLIh5JLIHWghQYlBNOzPJeH+Ng+ASMnoEV6j9ibCFbVQ7eQHGiNZeIxw50HzPv38a8BsZjpOlJzh1aOg3ju12YElwb/yr7a7yPVG6H4wyVRhGm+tiLeCeI5fYMRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YZusEDF1qzJX69Q76nZVilAAz+yoqKvuL8KWAZlvxXk=; b=jKHfch9xGvBMXXvQ0VGHF4DwyvyLVgarzo8HIIxAcTrEWBBFRWN2MT959BuM/yW/Ys547Dn3FjAq6GRZ9p0AJl9cYy7xaRnjGMhjpI5qp+lEGyLZFTbN+9vWveUfRP+HoesF9Hsf0mLEyrGBPO29W7vRgrezxmJj7Dhfcab5ew6ZW9gJvraJbH1zRd3U7+LPXF4yVFr5gxOzGBEvtR2royVqxcloVcb6xczn+EndJJ+L3Kr+hIGKvBfExtAc6TUbPI6x3yC7abWUg5CnCLpmrklaA3wG7hnpDHiottN17nYoA6TNfItOnW4ya18R5E9BTmfNeZqB2opgOeI89ysmnw== Received: from BN9PR03CA0297.namprd03.prod.outlook.com (2603:10b6:408:f5::32) by CY8PR12MB7514.namprd12.prod.outlook.com (2603:10b6:930:92::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:36 +0000 Received: from BL6PEPF00020E66.namprd04.prod.outlook.com (2603:10b6:408:f5:cafe::1d) by BN9PR03CA0297.outlook.office365.com (2603:10b6:408:f5::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.26 via Frontend Transport; Sun, 22 Sep 2024 12:50:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E66.mail.protection.outlook.com (10.167.249.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:36 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:24 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:23 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:22 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 06/29] nvkm/vgpu: set RMSetSriovMode when NVIDIA vGPU is enabled Date: Sun, 22 Sep 2024 05:49:28 -0700 Message-ID: <20240922124951.1946072-7-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E66:EE_|CY8PR12MB7514:EE_ X-MS-Office365-Filtering-Correlation-Id: f19cb38a-ff1d-4f0d-8f60-08dcdb052773 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014; X-Microsoft-Antispam-Message-Info: 5JSFFEzvKkQypFQpN5c43RPq8H5gUx+1VHdlEXmvkISXGGcEoSoeJuc7EFkQ9yJssI7oxmXTcXUzwXYOJ1gYfiNAkQAIgw35gYmxlzkEICFW8/YBQVC+pia546VQlo1lUN+wkSqAS+3I4INqTE/glZQ6IMWP8k8Y64F5crKzAmhVgq1R5SZOZixE+oMpOepS2axnD7Isb9L35Ln7v2iQTy6WNl0dnBZBVfk5aIXAemwE77gXFNeKHK1oZHo+fNUABufTEgC73qKxwigbjdKQRfxNxMUFjQpntgn/R3oqf9iWzE74Y6FGFyy/Pl3kd3dTrUwUIfy+z7Whb86zFVlyrWjooIRsaF6SdBizeegGTxjiCwZzSzlkw3DCuWjXd6aWv3jo26qW9UrhoAzA5LyLsb864ZskSAoHjYfjkf+AOMLsrqhu2SXr6HoO//WrOUFamWnQHBCDbe7pdq766KKkg490VpzbLUQj4Kg8ZCxQK0JMTfuzVH7MBATTbPpFPOJRrY52Bj2dtRCi98ohXi8Uxto2TXEAmUssTtlcm+ok7qDr3ZMk0XFIyd6NGyemuPkFAbg46w6tFjIeFUGjozOd72IUr6knG2uEqVEzkwkVDvQ5swZR/R/xsz2uvojGAZjbKX241wmo/fXIqUFHfAJ8KciSqC7oUxOpLEFgAarIDhTOnUWltsMCbkmFlAObZdGUvhgdCad0Bsd3oaPBquK8LZPhQUokhf+BtQB4eRV5X5c57erzA/jZpy+N+pq5y4snpdQ6cw21E57vYrKt30/TpQCextE0P6XN0S2fQ6afsVY/jgYUpuRSUhjQuc6IpxZnnvTTXMU8WGWxJweeUFHNxswhoSKyQj/XjbyiZzN13/KCyAKBiX7PDMBaGTKXZpJC3VXSPERKk3rb5oByHmjJHZqgBcNNWdsf1rKJsYZGWwqXXcJUJvT/XTyf2nSuFmy+9Mk92uEFvAPNEltEG9oPCdhLWpfXQ528Yo0LSXe75ILx+xwQLb+yLlS+pv6jYS/LIYiN3Gg+xKGTd68JuiP5CimmctloAa97fQOX6wLtp6uYj3uVZ1X/mpzJJevJK4R1JGPVzdyPICEmDmAAHiF4zKgvcZrTFmzMpJqDsrzd5rhBqy55WXc2IgGFCJsbRWxeMZDd7fUclL3b8aGDHPq7pDYtynuGTOZPUnvcMEb8+CSKkM73Y2ZpRi+ZFnaBISOVgGuwZK9GwsMuZP+XAXtEjgtvIOq2Wbav0Y0JwaVl2TzUMnz45UfKTfGS61Wl1RD3k6j5L3C6qszrxDJ8VjkCD5odN14lI7hIHP8l39cTylFkHhwaHUKcPJ48CwPPDr06lUuKmoNQrPBizL2eyQpC4jrHPoDhlabut3k9xCxFsc83fhBTDKK94eDZejvVpOpw X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:36.1097 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f19cb38a-ff1d-4f0d-8f60-08dcdb052773 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E66.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7514 The registry object "RMSetSriovMode" is required to be set when vGPU is enabled. Set "RMSetSriovMode" to 1 when nvkm is loading the GSP firmware and initialize the GSP registry objects, if vGPU is enabled. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index 49552d7df88f..a7db2a7880dd 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -1500,6 +1500,9 @@ r535_gsp_rpc_set_registry(struct nvkm_gsp *gsp) kfree(p); } + if (nvkm_vgpu_mgr_is_supported(gsp->subdev.device)) + add_registry_num(gsp, "RMSetSriovMode", 1); + rpc = nvkm_gsp_rpc_get(gsp, NV_VGPU_MSG_FUNCTION_SET_REGISTRY, gsp->registry_rpc_size); if (IS_ERR(rpc)) { ret = PTR_ERR(rpc); From patchwork Sun Sep 22 12:49:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809091 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2075.outbound.protection.outlook.com [40.107.223.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE56B2F37 for ; Sun, 22 Sep 2024 12:50:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; cv=fail; b=COGOJPVmMJ1XooObrAlL0sLox56u5+BAFzWZ2sYOz4yIVvy+cF3CtcpoUzND8tpYLuSRLfOA2ojvvn5MrBbAZfwGllDbIgIcIDihKvswQFe9rTcj+auiZ0KFC2u8Z09nl2NXB6BZEp/FH0PqR/HBqvA01/lcQRDXZBEN5ZQwNNo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009438; c=relaxed/simple; bh=UZqKIIMgTl16rdztC/8Qg9xASFhOKf5zdzk0F6+Yk58=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ie0E9dp1eSzP6u+E8ADJvIbuokZ0VzAgtMq366YKPbwhii65DXlqogrm+EviEY11UsmfeoEupy3a2w/RPG3Ahq3HU98asZ/aiNdfI9IZdfQ1pYlzjNKrOWHSI+hNrK0qs2Wo9GSPifv7SQq/pUWYRnsmAIQfvW3NcxTXrEo35Bw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ETTCLKLQ; arc=fail smtp.client-ip=40.107.223.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ETTCLKLQ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kSDel/5+XoQquJsDuKSWVq/D5OzrOARaFfzY5XhMM31pk4unpYkvX8E/wza7kHQ3+uv3SyZMRZ9UDo8zS6BsKpHa8yduFxAkVMF986dSJEclotq4K+zBFrFTLWyLz22lyxR5qhwwOHX7dJKiuZMBqiCiw91D6R5ske8ducc5qG5Gif1jCklIlDHYpbJYzdSP0Lh9yjsNW7W6bpjBCePaamjnQMUj/ggtVPK7dIS7FSXK+3TGyxMA5zOYWZz+w7SajsJz9EGt9FW2rCTTkFkAQ7Hy8uUfmNzcXiPZwJKZHHHdxb2w3aMcQs0nELj2TxkitP4UvohlqbSt25RSy6UcVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=atN0vQ70Ykwkr84OfTt4zIgErURFQak0rcSiUa4BZIQ=; b=C5ESgQRAA1kj/xM6/QNtClyJ2gUMKBckbY8SG+TlwVk/vnkcd4tlnvy1FjyMbrpIMqHedjXlirFiMBACBya4ePCP2nZXaXlxx26bsfmIkDSnGGufgJfr9rb1VAmMsXCJTU7uhxnZ5WMS6m5vlkg5XeSlqJCxOAqkfRnKDDYoUUZpoQGsFkSFVjSLQDjc7Ndjiq5x7QVVKxowz/UM7/1uDEJTDKLgmpzcLp2KqWlYxKFA0qk9cugQeQeDe4dxbHOfX28jv6yGfomZyAaAHUlt55qfSipDnjFLsjzxdA8wMHtUDWFmP4P7dvJgqJJtUySCMUdRvM3tXe2dnjkEixEVKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=atN0vQ70Ykwkr84OfTt4zIgErURFQak0rcSiUa4BZIQ=; b=ETTCLKLQzG8rb9UpTM5C0vUkcMthNO+TrHlH/xQymdAJDzhJN1fOsA6ZuXWEFuFWLEM1ppf1L0yzwOgle6wW6IUkyK0vj9ir9Dy4WZE5Y2AIdrN4oNKaUY+QNl/hKwqYi/7HIfdcHJCWbiPx/aAAGN9tOMeDDVHDz0inpsSznClWyWYx++yuLQvLJnzq0dPh8tm8Acj+IFic0CHtKY1QclwjDZ8XenMHLMRQKUcGqHVv0KqaJId9wcgJzksaB+OhMQBsL5A0E/BGp1cUr2nn+ClV8DiBVpmEKibCspXl6yVyXjxcJ3GEDl0+YUv1s2rrazBh73aOgqody8ZuR+h0NA== Received: from BYAPR02CA0044.namprd02.prod.outlook.com (2603:10b6:a03:54::21) by CY8PR12MB7267.namprd12.prod.outlook.com (2603:10b6:930:55::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:33 +0000 Received: from CO1PEPF000075F4.namprd03.prod.outlook.com (2603:10b6:a03:54:cafe::8d) by BYAPR02CA0044.outlook.office365.com (2603:10b6:a03:54::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F4.mail.protection.outlook.com (10.167.249.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:33 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:24 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:23 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:23 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 07/29] nvkm/gsp: add a notify handler for GSP event GPUACCT_PERFMON_UTIL_SAMPLES Date: Sun, 22 Sep 2024 05:49:29 -0700 Message-ID: <20240922124951.1946072-8-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F4:EE_|CY8PR12MB7267:EE_ X-MS-Office365-Filtering-Correlation-Id: 63bce7cb-faf2-4aae-f20c-08dcdb0525a9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: SJl2PdJKQFogIlDl2J9ktEDNklMK9wLfQA7FrRXC7gwh0Dyt0Nv08dFsg/wj33CLv2DFrtAmj2X2PUqwsyTx4gYupUxWDYYJpFzgg3EFfbZ9fevuMGHSH2r1YjpkAT7ef6xwrIO1e9HJuX/sgp6ktfi+LwXPE8GiAx1/zwfogajNOHuFC6DaHVDxrsOeRBBSeTQ022Z1gKEd0dju81RW5VJ+melB1DMaGhqijgu8Dp0ZJSbgLusFG3vXTUnT9Oo1jxvAxHqpGPjCzYg3C9stChq8aVFII8suLthD4ofVI3wTzvCfA/f2wlveuFVgfbnJmkOARV9YJ6NdXJU09pnYU0cyhMFHm3kli2nbdbpCJeT1VRViBcrKGUA9j/1nU8OaVLQEGrRHEocSXD+VpbcRHRgiIE3vKusqjh5E0Nz0lyQk2/YBRvdlzHva8FY9qu6VB7aUY61+Wz2xdygYCink+F9gqW5B5eaTjowKFWIG63Hgcu3N3+fk4nvX5iatRKEjWqjq/zY7BetGCRZInlifXRyRWPvk7Sik8Hg/ubdHek8ChdtqmTKZieDCRaK0L3zn51Obop8/wxIRyP0QXrgAZRUqn8vCSdtEH9piAkL+JfUnjPlZrMultHQLS3rswcsHfhQlr4/UCOxlhSsEIDjCeyftjkVBL9YZ4CGOvvi5KIfZI+mCIVpw+W6lQ62BJVWTF7vkqmhlc5WSST5kaG27OVV7HWIAxz3kdbz4xdMtVz9hPzgaNnU+1BVnAXN+NCv42z9Y4T3M37MxxrkGlKIfuyMrBJsAmNwM4t98ZC0L+pUP1dVJWJcAMgFVirg4qOlN3Qcg+tgoIbamFkDMR/Za7uRmnk1Ka9RkVVWZlzzHg9HYHroYIlc9Q5RDsQ4UqEBQBsJUcd/zijl88TZbn04yfAUazWiAH8VnSbJP1XaqE7DDb2S/2sWRi8UFnnYcG9sje+T9PTi6HtdDLA8hMgDWWmEj02uB8fpoI5PCL4e2EQgN08L+kN0fzpGboYlUYqEU0xL9SLUnslSKyjzsUTnyJSb8A/yiwl0Jbc3dMmu4fIQdm/wPoQYY+2zaNBzEjuGtCoAozd5NwnclrkCCaGetzjmkpolVPG5ixulPWUohwButryaxKoOHCTszz51DhC7nY9LChTZPwE+ZtQhpUFQfSXjwUwgH+vWyVhQsU7sHp1131WqM3EFCWdVAGQDhkDpK6U+tKX4HQwqpmnTO5DoentkHPu2Q67PWvnY9ivh0u53ZCTO868WrAUz7qkhdw9SdyJZe5vicWDc1EqQfKwQk5Sbq1RwFUYC6JqAU0b29HYxfM8B6tqIETv7V7+vNkW+KCOr/fWScugL0+RrLUEyqwanJHHcIijAZ+kiCP1xmJt4= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:33.2044 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 63bce7cb-faf2-4aae-f20c-08dcdb0525a9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7267 GSP firmware will periodically send GPUACCT_PERFMON_UTIL_SAMPLES event when vGPU is enabled. nvkm keeps dumping the entire GSP message queue in dmesg when receives an unknown GSP message, which is too nosiy. Add a notify handler to prevent nvkm from dumping the noisy GSP message due to the unknown GSP message. Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c index a7db2a7880dd..3eea6ccb6bd2 100644 --- a/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/subdev/gsp/r535.c @@ -2628,6 +2628,7 @@ r535_gsp_oneinit(struct nvkm_gsp *gsp) r535_gsp_msg_ntfy_add(gsp, NV_VGPU_MSG_EVENT_PERF_BRIDGELESS_INFO_UPDATE, NULL, NULL); r535_gsp_msg_ntfy_add(gsp, NV_VGPU_MSG_EVENT_UCODE_LIBOS_PRINT, NULL, NULL); r535_gsp_msg_ntfy_add(gsp, NV_VGPU_MSG_EVENT_GSP_SEND_USER_SHARED_DATA, NULL, NULL); + r535_gsp_msg_ntfy_add(gsp, NV_VGPU_MSG_EVENT_GPUACCT_PERFMON_UTIL_SAMPLES, NULL, NULL); ret = r535_gsp_rm_boot_ctor(gsp); if (ret) return ret; From patchwork Sun Sep 22 12:49:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809095 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2041.outbound.protection.outlook.com [40.107.244.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDBA54A3F for ; Sun, 22 Sep 2024 12:50:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; cv=fail; b=O3hibrY8FruIuaY24wEFiZgIqkkcECZkX/VyJIZCMpbePa0toxFdQG8d3IFt0y8gN5g+UYDKB7Q3fOnp/ijk3OKA14ej/vS+lTjBWXFk8nGoVb1VP0Sek08CZguN7F0qbbHGjSSHM/e2/W7t8RbwkuwWE6oYCIiCX7Vi/JAHBoA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009441; c=relaxed/simple; bh=jkeJBaIZqIh6OVemI6P9Vs1IIGbmNkHrb63mM9/7MJk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=otMN9igtG6MxziMPPID40p4bVPmtXidKo7ispXTARejuxZmu6TNS9K901tQfFeBlJsuLNM531Q+MXHovHkix0eux0MQz4R6EBle25PtJsCzefMn+0lvMes3+F8/s+P+nQUICyhP9fn7F+5wMnUX3LLnzVN2Uy2Y42rnBonqO8GI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=oK2L0svn; arc=fail smtp.client-ip=40.107.244.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="oK2L0svn" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cI8vmGWJYcCUG67OvDsO3/TunNth/uGBkFxTRpIttvoeyhl5eFTbiUqZXahUvDsyFoukQCaJ5M91dOPnxRRx7YSaLV6nIZCwYzwiRoEPmJwCgQtS9vHQDR3QvrWK0EYYH2A2gjeY7Y3WY0xCpo2l8ZYa9RKsTdFNWskq/nmd98I9PGx3PJ6yha2uzHaTJB7c9kZ8VNz2gdl1X/itddA3gfsYAEaFmXyRs60SLVlfh2R3wkMmxbp82TsOLl4DCsg88jWX/v6hym9eTaysTp4hrqv6zYclJcjsOERelY1yky4Zj1vPO1NDqHIGF3cXeAU+dvh/yg+55ODcsURbRJnQXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Duc8HA8IkbkhMF79mEpikrfQ1/QCznCR3/5vRo0Va4A=; b=Pg8QRWnPMkduNCuEq4KtZthG/gLzCjjiKBlHZHzXQyPXlXG9RpdJnCnT+a5SuTgopZx3Gce3S3vBjJDTFZyRwRBmk3JCEGnU2Xj+U0HQ+jh9bh5Xq5TNDiRQ9o9MmzN3lnmtBBaQzfqsd2WBFjl3/p2wzEtsL8FRalLSlOr/RxCU8/nx3yC1OVTIXWfRjsWaDeY0IANMqXxAVZiNW5IZT7GkfmrFoIb6dbiXhc3Ap9MtqZ/EKQz7uEo9RtiAl8TpxJRf68B9+oYZjx2yG0+HzKnMmO0X7xZ1YEV9oBXctRlLiW7laoPLvGyci8+S5MxppLVCtmy090Dpilpti4hKaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Duc8HA8IkbkhMF79mEpikrfQ1/QCznCR3/5vRo0Va4A=; b=oK2L0svnEFsGhoBD1/LweD0uBctyQcBAPOxcWlcTl7vWREHN1Jz/EJ6qYv6QwmD951SA4jNUNzLpDIR/87jVEyjMD0+ewhEqohvT9EWbJ0gXtneij7Jgb+LqHKEv+OgDD8NyP4JSMfPNREb9H6kNx4uYeh6BIBHGF1NPKKBeJLLUm2LE2MZrf5eHQbe0mJlGozVXBzbacA+dJaYZafKFylE5ntwErJ7mPZNPVXJqdfvPE74RSXW1JIKSsyBpItaZ14ctLc3Tiab//RzSzbr9nCD4NF0QDL/NPMY6nP6o3ljVVC61AZ73+kdMMcg1TUaxjwmgTc/95y8wUyuCNtmkQg== Received: from BYAPR02CA0051.namprd02.prod.outlook.com (2603:10b6:a03:54::28) by PH7PR12MB8054.namprd12.prod.outlook.com (2603:10b6:510:27f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Sun, 22 Sep 2024 12:50:34 +0000 Received: from CO1PEPF000075F4.namprd03.prod.outlook.com (2603:10b6:a03:54:cafe::e6) by BYAPR02CA0051.outlook.office365.com (2603:10b6:a03:54::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F4.mail.protection.outlook.com (10.167.249.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:34 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:25 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:24 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:24 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 08/29] nvkm/vgpu: get the size VMMU segment from GSP firmware Date: Sun, 22 Sep 2024 05:49:30 -0700 Message-ID: <20240922124951.1946072-9-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F4:EE_|PH7PR12MB8054:EE_ X-MS-Office365-Filtering-Correlation-Id: ff50ec71-dad8-42c8-107b-08dcdb05266b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: uk6pmIclN7MROsrQaKUtkrRmnWbjtN4VencYq4it302uuGKAbQaXwY6oR2erShGBGfd1KDO6J7uXgqa7tvPggCAvXp/DiXTeyqcIjgJxmcLkftTbj/tyC/AjcWsE4IU1yxt+yT+QcTLHL1tHe9iu4qR4DYiUSVYgv1R0hNQoWHk1U0ZEV6D3xbIxVvqH/Yl/J569pskavz6irQ29aAyQjYXsoePXj2BCQUK5katkOFIXAjVRpFdK86RwGEs5kYU8CtQJ/PwzhfRv/hyvXuOF1HGqWez9/V9bwNyJ+KBKnUaRXgsSsH+mxzRZLkhCF5Q4bxFNDbs+AgDjHSZztS9eg7ZCdpo9G0QCHit+RwbJWIzMVVvMXZZB1yVMla04tSV5r+J9EzYKeUQbPHyO0b/LAbXhdeC8bje3zoPhbIL+fqxFONtFhdM3ebio+tix+76kP0l7EzgPFUYwVHmCazg2BBPgZelu76S0oL4i28rBWukG7YjiqrROWujdOiLCBwxkS3fpjoY//2bonvWOuxRSw0tOiF9XfAZ1YYgaGnrfjPAL4ulkyhxlIda05Gjx9Zj+VpghVOIsD5goLPrtszWVKIEay06B1QFq2ej7amoYKXGr6o5pBs5O8XsBH8lKLVsU2O0/uuksf8kPQmXnxQofLbcEmJj04s8M2PVzEUjCuk+4FGUgg6CXnFwFzyOmXfZnj/CStLYeksWAFHMQrlGJhA2ilpQ6ay95S/khp+k+Rkm1sFFxxGHBPy1WPGPSNqDAjP0ITQmxTWJCiXBC0z0X4frdo1n4VPJvOuitTVtXyd2EOZzNzIHT5J295kSEp5jLUorqcnEU+e+VNNggu+6UI+fx12TzzxHW54IkSoYriiZs94CxEVcMaPgViuCXIpIg5pLrKPj2807N/FaFnK1gGcvz05sPz9f5YLRIcxeDy4RHaZGNjdWLIKCrQcokzsYEwmx8na1j1sdNqRYNyK+0120IGwLgcYJbscXrVL1zHg0KSMSiYXFBl6j1fwChFp0fKrgDblnU73EtdDrzOpWLeizq6G0thu45J+lebS/t08mznTpSvtM73i+SKiK7sUyuaNjIt6kv9HPjTsGEsqKc6EDhA871snITXV0mFgfmduj1NrKBmYch1T2dJAZI1pCKnAzY37JMGzkagaP6+XwCI8dpH7qk9cCinN48hnSObyVJoFLjoAGb8DFuDirMn1KY3StfF8/6IdWn1CUkokGX5Vxr2bYDOIK+0Ao+v1Z+TdG8dIEu8kW87G6Zuez12n277bPpORAgKBNgpKJZngKe9q9A5SXlTm1b5FpKhwTWlAGwQn6Z6iU4krbkE59WDGOq62Z9CvLcSAd7dG67QJNk6DfL4cx+dPQU1/HyoyXInrJ6XnnY2+7XPG2pXar8BcKbJiYv95PEkmi5CtNEL/94oyEZF3r3ly+UD/kgmnUvitCdpqyHPXOJb9ciP3cYxgJG X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:34.4700 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff50ec71-dad8-42c8-107b-08dcdb05266b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8054 The allocation of FBMEM for vGPUs requires to be aligned with the size of VMMU segment. Before reserving the FBMEM for vGPUs, the size of VMMU segment must be known. Send a GSP RM control to get VMMU segment size from GSP firmware in vGPU support initalization. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 2 ++ .../nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h | 12 ++++++++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 30 +++++++++++++++++++ 3 files changed, 44 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index 6bc10fa40cde..aaba6d9a88b4 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -15,6 +15,8 @@ struct nvkm_vgpu_mgr { const struct nvif_device_impl *dev_impl; struct nvif_device_priv *dev_priv; + + u64 vmmu_segment_size; }; bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); diff --git a/drivers/gpu/drm/nouveau/include/nvrm/535.113.01/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h b/drivers/gpu/drm/nouveau/include/nvrm/535.113.01/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h index 29d7a1052142..4d57d8664ee5 100644 --- a/drivers/gpu/drm/nouveau/include/nvrm/535.113.01/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h +++ b/drivers/gpu/drm/nouveau/include/nvrm/535.113.01/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h @@ -97,4 +97,16 @@ typedef struct NV2080_CTRL_GPU_GET_GID_INFO_PARAMS { NvU8 data[NV2080_GPU_MAX_GID_LENGTH]; } NV2080_CTRL_GPU_GET_GID_INFO_PARAMS; +#define NV2080_CTRL_CMD_GPU_GET_VMMU_SEGMENT_SIZE (0x2080017e) + +typedef struct NV2080_CTRL_GPU_GET_VMMU_SEGMENT_SIZE_PARAMS { + NV_DECLARE_ALIGNED(NvU64 vmmuSegmentSize, 8); +} NV2080_CTRL_GPU_GET_VMMU_SEGMENT_SIZE_PARAMS; + +#define NV2080_CTRL_GPU_VMMU_SEGMENT_SIZE_32MB 0x02000000U +#define NV2080_CTRL_GPU_VMMU_SEGMENT_SIZE_64MB 0x04000000U +#define NV2080_CTRL_GPU_VMMU_SEGMENT_SIZE_128MB 0x08000000U +#define NV2080_CTRL_GPU_VMMU_SEGMENT_SIZE_256MB 0x10000000U +#define NV2080_CTRL_GPU_VMMU_SEGMENT_SIZE_512MB 0x20000000U + #endif diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index d6ddb1f02275..d2ea5a07cbfc 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -4,6 +4,8 @@ #include #include +#include + #include #include @@ -86,6 +88,26 @@ static int attach_nvkm(struct nvkm_vgpu_mgr *vgpu_mgr) return ret; } +static int get_vmmu_segment_size(struct nvkm_vgpu_mgr *mgr) +{ + struct nvkm_device *device = mgr->nvkm_dev; + struct nvkm_gsp *gsp = device->gsp; + NV2080_CTRL_GPU_GET_VMMU_SEGMENT_SIZE_PARAMS *ctrl; + + ctrl = nvkm_gsp_rm_ctrl_rd(&gsp->internal.device.subdevice, + NV2080_CTRL_CMD_GPU_GET_VMMU_SEGMENT_SIZE, + sizeof(*ctrl)); + if (IS_ERR(ctrl)) + return PTR_ERR(ctrl); + + nvdev_debug(device, "VMMU segment size: %llx\n", ctrl->vmmuSegmentSize); + + mgr->vmmu_segment_size = ctrl->vmmuSegmentSize; + + nvkm_gsp_rm_ctrl_done(&gsp->internal.device.subdevice, ctrl); + return 0; +} + /** * nvkm_vgpu_mgr_init - Initialize the vGPU manager support * @device: the nvkm_device pointer @@ -106,11 +128,19 @@ int nvkm_vgpu_mgr_init(struct nvkm_device *device) if (ret) return ret; + ret = get_vmmu_segment_size(vgpu_mgr); + if (ret) + goto err_get_vmmu_seg_size; + vgpu_mgr->enabled = true; pci_info(nvkm_to_pdev(device), "NVIDIA vGPU mananger support is enabled.\n"); return 0; + +err_get_vmmu_seg_size: + detach_nvkm(vgpu_mgr); + return ret; } /** From patchwork Sun Sep 22 12:49:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809097 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2066.outbound.protection.outlook.com [40.107.244.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7346111C83 for ; Sun, 22 Sep 2024 12:50:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009444; cv=fail; b=aHyfgHbduR1ym7mxavBGPMS2sbvi5FOL7F9aKaCJ4bqo6xScRY3InUgjWLcT43g7N5E2uVXfMxbl4wp87oPhYzdHMw+Tpxky1LxPVat9jXuaM9m0YuWpMJJrXMTSRdlUPFAQyzNbJPcZU6zmiVaYFm+bkq5/CvLJDQr550qEICA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009444; c=relaxed/simple; bh=Qc/AnBRxZy3jRHJcQjPNk7fWk8ULQqZHo1LMo1YVgXw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=if63rNjJy56LKJUTMb3Zgoy8vZLUrTEH0AHEFA48wAjCzcbKJzJrjfCF0aB1qjxyzgkZcQ7RLjVVX0E4aCMYqfPZcfdh1WfVLdiEBdv4g5+x5UHx4qkBmNP18t51hoK98YBdYEqp4RT0J4+/t26/EbkPCFz1uqVkR+erRrdgCZA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Qkc2tjEG; arc=fail smtp.client-ip=40.107.244.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Qkc2tjEG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Nvs7OFzYaSQk9o6n7nYhRP2zP70HaxvtXOCgHpKjUYazRmNaCozvgW8/YaHXoSuEWYCiAwVubMOgzpWSt4ZWl01c4Ew9Nivg2/cru7Udv5Z56MHd4l4/bUl/TEgfZGtFSYXVi6nJJi0qy1rhCLdFjGlZ/xlBTIoXAHDiXNCpRqaAmB67m0UqQHs00QOxYBhV6BAEa0WVATKXcRJU0h+5DzMsG+RDRIzYstYMCsublWrAxOzJ/GHRHkceUuj9Bkpi8Of8/SFlNsAGRSpu/uwSNOuckLG2S1Pf2oWXwCSko2DyeSiKU9+BtlYylu7sAtzMyTtZ+De4l0Y5pziCK8T4qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2vAXGW5UV/Bo3oTah6m0M6cPCsVTL0kinpOCmvs4hZw=; b=rrSpznPubRPYxq7Vj7aV6sVRwGbKZ+6Xw9031jWDxhNQ9+p9cCo+5OBCw0cUPaBUlEhDUmoHwcolvue9uc9xo4v6aMs5SmiJmpfp+spvP3aavMSCT1I5sDC5bTTR7l5x7eBfUyKk3JFmtGN7vO7K/STVb8suIfXTU1kzzRcG0LPKV4pbKu0U3JEgzNOAqbPse1O8znZ6mJ+oIm1XFC9J8riYRt2kVaVpvAdfEzqKzHVv9JdfGJ47RtmiFw2fEWXO4gdBoQEdrbjIxfddx2dXcGB6uR7rB14BEdixfQ6RXvCtdrPcJR/wNDh41na1l4tZ8L1A+o+fHPYkSdvQW76cbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2vAXGW5UV/Bo3oTah6m0M6cPCsVTL0kinpOCmvs4hZw=; b=Qkc2tjEGMW/OZw5AQ91yvUq0fikkIourXM4LCFqKQ+eHaneUhdpMfg3JhzniL5vgWfctycCx0lu9ve09l+mnpUTb6KgovqYMbbQu0BPAV4HPyGV5VshFCg7l2s4yXoOBhQ2EiqOIVypYtyZgEkFZJxHJTNVQ3qZLXZUai/4uwVrhSNgO3sQsPUGGGj4fJer/8kDg9jT+zk+tPGf0ENILdxn778ptxCxsuao02Yioum6SqaD4luhpyZ/jQu9UZJdSM1XKWG5usuUK+T8L6RYaWgk9IBWNZmLpSIm+RmFdtYnB2C7RY+WqFxhgyrB9JuPGxXoIBNd1qYA+f4sgk8wrBA== Received: from MW4PR04CA0329.namprd04.prod.outlook.com (2603:10b6:303:82::34) by MN0PR12MB6176.namprd12.prod.outlook.com (2603:10b6:208:3c3::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Sun, 22 Sep 2024 12:50:36 +0000 Received: from CO1PEPF000075ED.namprd03.prod.outlook.com (2603:10b6:303:82:cafe::d3) by MW4PR04CA0329.outlook.office365.com (2603:10b6:303:82::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075ED.mail.protection.outlook.com (10.167.249.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:35 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:25 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:25 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:25 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 09/29] nvkm/vgpu: introduce the reserved channel allocator Date: Sun, 22 Sep 2024 05:49:31 -0700 Message-ID: <20240922124951.1946072-10-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075ED:EE_|MN0PR12MB6176:EE_ X-MS-Office365-Filtering-Correlation-Id: bd1abda6-538a-47f4-d76b-08dcdb052721 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: 8QGZz4p4dzFDlKIaQ2ZQBOwhpbhVdRY//wzAgCz+jxDR8LFVhre/HbHQ6RLOdhfE0QcQwjCH78LrPFlbb/pgQh+P1vtO/fuOYiLUfeNNRSAi2y1J/qNhydDidbiOHkgWuyKou85VdpLaRyYIERDh9OcLRy4zoXrt5xfvjGP1X91osoUmxzEg1YlYVynaPnlT/xSRy1UQqTA2NyL/r5UEWRwcGgox9PTJVuDLm3nNq+6jIX3xnQgLDMXXWMnuEl6hLBtCHVBkkmO9AKA4OGhildlpZozfSzKBPTGI6fBmW8T3F+LFF2CYNMDEU8XofddJVXHQzTgezNmFxpoqssM59B/2zoQQqGlXtE+yYVXFA9nRs0QhBvVcN9s0TtQ+Skb3A6wB+s8SU4PsvqGxIyjiDiTSVeNX2moATaX2SOkF0E1sxKmih3b/ZQ0M9xDI5GRRbwahuBscLu10PAZFZnlCMVFURF+EfCMA6wnm3DctX7ghiDeb8E7tobxGwZYUlZsTE++jJoMhVWq1IFdaAzNbbgc6EzUilz9CR/9xlr8446tQWv7TBoSJTD1vEX898XRJjVJr0jJ2QPBHevWNz0kl50mdzzSbm4HNM7jvMJhU6Rm1E9Mo/a4XyaXmCfwk+2JkAV0FkePNArdjCuG2vQLRpdEMVylUkPqXMtd6UIYhS6oc4CEnbtqheTF2rG6GZLh2Y73LCVECdtPw7FQsLQmM9ajYYmL8aZxMzZN481Iw62nz2qot0O6urdKh96ko3CKYMGwz+30rMiyXJ+HcjzWgZZCRi5CW+hpiqmINDirSJswIRKrvMgb7HCKL+KpmLGdzQwSZGp+1k0+15xOKQzKM1b+gj6obdueW3hJ3HKHSX0fpUYwZpoPmapJQ23NTsFp82M3GdmgGYbIfw2XS3cssJmasBQBfXKd4xpQ5Q4Eww81sTGgs48a/uOC0wJz733O7ZFRgj+ih2Y8sBnB6Q0x0LJEsGhjZAOhScJNw5oY4CHSy9SXPp+eB/LT0QjUJMHDUuKMV3wUi5al7uNcBrj9Ej74STx2vTI+beiq9WTPO2YXaaGRykxH+lYTIkHjFsapQjQaSxYDOBvjX8PRBB7O0L43xi8ysEUa4Wf2yZx3fBlhNFTC6VZ+aqMecjF0ZoUCp4HnKRdzgBcXyUq5pZpHl7yimAi4PM0fdRCO58swFUMjrd10lkvLM/SBI5A5eQNw418W5htIz6er6ruKjWwjrHElbDAW7B/TALHDbl/OuwBJX/EpB+GlYYYtxJwLZKM8oL2S4pp4oENn2+ZphtyQW3HoMlfYl2rTOjI0v7JKHx6MmTLbbjzLVL6sOONhqEobQSRJ8VpIN9r64ZVpUadfqF6YPORgLxwhZflHWQrXRY2Vh1ZRCgpH6PHy2owZlpOJ6MLn3XzOnXhPk2yQdHF2GXqjJh/FKvu2iwjHpIKcRAElaLuc9V0d6JcGDYbrYt8N8 X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:35.6835 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bd1abda6-538a-47f4-d76b-08dcdb052721 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075ED.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6176 Creating a vGPU requires a dedicated portion of the channels. As nvkm manages all the channels, the vGPU host needs to reserve the channels from nvkm when vGPU is enabled, and allocate the reserved channels from the reserved channel pool when creating vGPUs. Introduce a simple reserved channel allocator. Reserve 1536 channels for vGPUs from nvkm and leave 512 CHIDs for nvkm when vGPU is enabled. Signed-off-by: Zhi Wang --- .../gpu/drm/nouveau/nvkm/engine/fifo/chid.c | 49 ++++++++++++++++++- .../gpu/drm/nouveau/nvkm/engine/fifo/chid.h | 4 ++ .../gpu/drm/nouveau/nvkm/engine/fifo/r535.c | 3 ++ 3 files changed, 55 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.c b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.c index 23944d95efd5..0328ee9386d4 100644 --- a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.c +++ b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.c @@ -89,13 +89,14 @@ nvkm_chid_new(const struct nvkm_event_func *func, struct nvkm_subdev *subdev, struct nvkm_chid *chid; int id; - if (!(chid = *pchid = kzalloc(struct_size(chid, used, nr), GFP_KERNEL))) + if (!(chid = *pchid = kzalloc(struct_size(chid, used, 2 * nr), GFP_KERNEL))) return -ENOMEM; kref_init(&chid->kref); chid->nr = nr; chid->mask = chid->nr - 1; spin_lock_init(&chid->lock); + chid->reserved = chid->used + nr; if (!(chid->data = kvzalloc(sizeof(*chid->data) * nr, GFP_KERNEL))) { nvkm_chid_unref(pchid); @@ -109,3 +110,49 @@ nvkm_chid_new(const struct nvkm_event_func *func, struct nvkm_subdev *subdev, return nvkm_event_init(func, subdev, 1, nr, &chid->event); } + +void +nvkm_chid_reserved_free(struct nvkm_chid *chid, int first, int count) +{ + int id; + + for (id = first; id < count; id++) + __clear_bit(id, chid->reserved); +} + +int +nvkm_chid_reserved_alloc(struct nvkm_chid *chid, int count) +{ + int id, start, end; + + start = end = 0; + + while (start != chid->nr) { + start = find_next_zero_bit(chid->reserved, chid->nr, end); + end = find_next_bit(chid->reserved, chid->nr, start); + + if (end - start >= count) { + for (id = start; id < start + count; id++) + __set_bit(id, chid->reserved); + return start; + } + } + + return -1; +} + +void +nvkm_chid_reserve(struct nvkm_chid *chid, int first, int count) +{ + int id; + + if (WARN_ON(first + count - 1 >= chid->nr)) + return; + + for (id = 0; id < first; id++) + __set_bit(id, chid->reserved); + for (id = first + count; id < chid->nr; id++) + __set_bit(id, chid->reserved); + for (id = first; id < count; id++) + __set_bit(id, chid->used); +} diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h index 2a42efb18401..b9e507af6725 100644 --- a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h +++ b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h @@ -13,6 +13,7 @@ struct nvkm_chid { void **data; spinlock_t lock; + unsigned long *reserved; unsigned long used[]; }; @@ -22,4 +23,7 @@ struct nvkm_chid *nvkm_chid_ref(struct nvkm_chid *); void nvkm_chid_unref(struct nvkm_chid **); int nvkm_chid_get(struct nvkm_chid *, void *data); void nvkm_chid_put(struct nvkm_chid *, int id, spinlock_t *data_lock); +int nvkm_chid_reserved_alloc(struct nvkm_chid *chid, int count); +void nvkm_chid_reserved_free(struct nvkm_chid *chid, int first, int count); +void nvkm_chid_reserve(struct nvkm_chid *chid, int first, int count); #endif diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/r535.c b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/r535.c index 3454c7d29502..4c18dc1060fc 100644 --- a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/r535.c +++ b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/r535.c @@ -548,6 +548,9 @@ r535_fifo_runl_ctor(struct nvkm_fifo *fifo) (ret = nvkm_chid_new(&nvkm_chan_event, subdev, chids, 0, chids, &fifo->chid))) return ret; + if (nvkm_vgpu_mgr_is_supported(subdev->device)) + nvkm_chid_reserve(fifo->chid, 512, 1536); + ctrl = nvkm_gsp_rm_ctrl_rd(&gsp->internal.device.subdevice, NV2080_CTRL_CMD_FIFO_GET_DEVICE_INFO_TABLE, sizeof(*ctrl)); if (WARN_ON(IS_ERR(ctrl))) From patchwork Sun Sep 22 12:49:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809099 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2077.outbound.protection.outlook.com [40.107.243.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 863421803A for ; Sun, 22 Sep 2024 12:50:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009446; cv=fail; b=mmyK7FzEje1OrBPI8HRF1APP7w2vLNb+K8XVEfMbCpzi/mkRb4sat/2zEOMbB5ZNQFLdW55Ga9DbOll138wZ5EZQAUNzx7l5bFirsJqYf98fU0Mo80wvO5c64yFJkVyKEUxPsrDnaKN6t+ZVZoWBhtZEeIWMdXHVdnh2ppuLyig= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009446; c=relaxed/simple; bh=j6wElKvtIHrhSp3eMA7dRaUbfxRhsAbJBjJI6kyF5nk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=S1CiPywuz0xg8PhxgQ4It2xUuXdpRPlaU73o+rAex/1WX28f71PYDATg3Sn8aI7MCigy3ACECsuZgRIdQWTOULYcAGIGr7M1g3luF8L918sBupKBP5pBft5apXh1iGxa1uM0e8zgQyCAebNh/UQEXWUh6Ne4xcbkFrpUAKF7SJk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=urLSNysR; arc=fail smtp.client-ip=40.107.243.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="urLSNysR" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uZ26g37LWJ4P5OnHErB/HYtf8eT+qczoSzwfcG6ynXotbEaOGjLGN1eqsvhiQqb5wfNNmWcYCcJi7S8JkxEseD9RRrcmFfJRIFhzUo2uWCoQ8F18W7BAx7yjttlTU8NEEoRPe+4VHONVEAF69/wuBHqBxMaI1Xwts81yw4qttTK7DnEvFw1TbaUq8+o7h2DUlsIUYar2faOgbfbHYz4QGmPTu9YIbeVhHg9c4IylRjI7IYcuaxTVVZphcDV+MmrE7eLpbFD26aT9MKIer3ad7wNMLVklUmdtQBpzsbGO1EmugDdVxEaBFMM7LkFw8ulDDBrkll3MIOSpzqokP3Q4dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RHIf73+KXgRjudQBx+lRNzy1c61VRJCJpBw6rge7CKI=; b=yct0yyqxcLdq8Nr8D5Dhr55coEV9Ij4lGdwlDBDQYI21caKXVJoJmNBUkCW2ZUyjElNS+GtZygWIGoHCjHpsNFbwHUWB4a5BJ1t6eRQeJkoxWhDrcu2ctEek6NlaUSZGegHCYvDvS8Su71bTNYZPA/57ol+mk90kiTv6MSuxdjhi9AwBrNeSZe8Sc4MjoDYcJNTAbguoqKiqhni482a7+9koPN6qrwPKe5D5prp++vFIfaABgkuMUVNhRPrAn2WR1gn1hSLpOgb+pwTqhnGEixzRRKGaWGN/MIOgTzFGCqwgqzYDmH8VpHwBecneJ5f6MknMPYWLLwarKEoIKOFAqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RHIf73+KXgRjudQBx+lRNzy1c61VRJCJpBw6rge7CKI=; b=urLSNysRQ1XsJzQy33eJ/eFwH8T2hR7nFJTJx+UMwLwFF6pxMGK/ulvpzj+YbG3bulBRLiIYDAOPv55h71t1qkEzdR1nmS1O91t78Eh0sFmQPyC4SB1J7RdelNs/cV5o4ANbG9/IvPAbzrYn4VIFp5UDx+tG7dRgm9FpCCPJUFWaqw+YnRTunvyquO47ClvuF6MbD3qMnhjemZJB2f54ztdihRAzKXg0W0gK+OXbB+fhukb2ukt5jilTC7kwiKJaZeBIIPHTuykqfh2TTu8l0FjgG79fQXodi/V0dbNO0gXyf9E3OI5Xu0p1zZnUx5chUhsxIV6gS/5t/hjMBu8AIA== Received: from BN0PR04CA0130.namprd04.prod.outlook.com (2603:10b6:408:ed::15) by MN0PR12MB6320.namprd12.prod.outlook.com (2603:10b6:208:3d3::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:38 +0000 Received: from BL6PEPF00020E62.namprd04.prod.outlook.com (2603:10b6:408:ed:cafe::a2) by BN0PR04CA0130.outlook.office365.com (2603:10b6:408:ed::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E62.mail.protection.outlook.com (10.167.249.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:38 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:26 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:26 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:25 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 10/29] nvkm/vgpu: introduce interfaces for NVIDIA vGPU VFIO module Date: Sun, 22 Sep 2024 05:49:32 -0700 Message-ID: <20240922124951.1946072-11-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E62:EE_|MN0PR12MB6320:EE_ X-MS-Office365-Filtering-Correlation-Id: ab166165-3f6e-4c6a-da17-08dcdb0528b8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?q?ynVgLxY+rJMd69jc0libkDY0lmR9SU/?= =?utf-8?q?HuB7zbiKPznybGmIMKv/RcJSDkB+Kd7qYkEdAGnPksurbcEGTj/RDb46nPf3HeQSK?= =?utf-8?q?GeNWDl0LrvhMRNhBd9YP+ytDeevFo0g9fCyUwy7x9BkPw51XPqrrmbfWCKE3xx83b?= =?utf-8?q?E3Prz6+abBRe1A54dFdzVt991aKznXE0+sSV3hR4zHyoRDgiyHYa8sMm+kOaDNFPF?= =?utf-8?q?AZI0jV0WLJFirxE+GzJz3AUXfzH2pXpsVMNENheIBqVfhCaJAM4s26DT1ookrptcG?= =?utf-8?q?K3uBXTdCAfcnX+uEjbkGXj5nG4Q9P8JR7SB9LRD8MuWiOYfL53JcAJavvV3YWJaPQ?= =?utf-8?q?JcFKX9teWMzsyhlxVEMrO/Nu3gRhJbUvUbtimEW3+IjYHJS/OPXqtc/z4rsqPAScI?= =?utf-8?q?tOAJpJAUaizY6EybccEgKn5atpWcWKd2oiL97slKWVa5RMFRSw1SSxNEen46OlE4J?= =?utf-8?q?g4uHLT1bq1kfG8CD0iwcqLOI2rRKrtQKBU3+QUuZo6gbaw/HvVEhR1Z/kbnp9RMZc?= =?utf-8?q?tbPhUMq0Eb4mkpDjRjJUrd6iwtKiIodEDDKypct2N8v4rpVAjn6Poa81KmNlsIAY3?= =?utf-8?q?dWnDNpigLiZMNiMJxAIZbgIlRSMCuFPwoHrIRT+QXNc0UC6DYZSJA/rCAxWIcFhoM?= =?utf-8?q?2qdDM872B23cVblESZ0/xcptdx/hFN9fYlzAY2Hhhhndb6ibhZb6euDhlbQYhIfjf?= =?utf-8?q?tpfYLpk4WsZeTeccZPKvcDcw5BYBzl8Tr0F419ltQ85yKCj27X1hqaoHibVUdJxLZ?= =?utf-8?q?dnQ8OVlFH9erLozG4Ql/tfXmLDorV0bp4dyTL339CrTlVivhZ9O9/O6V480Kd8gI/?= =?utf-8?q?aO40cPWDzWRAKEvyXYIBTu7vf8O6j04QiVc5UbZw9/XfGFd+VHTOLB9C6jnya37dk?= =?utf-8?q?9c/idwQaz++XdJKcr8c+1NkZPDlsGIQag8sQHaKyqBmi9GnLeNELtxdGnKXAUtBIC?= =?utf-8?q?q4QmGLiYtYMxC0ImmCn22Z5rrE4wdKoMRNoCVY2Psorlye9ARj7EwFUlC3OdFlYxv?= =?utf-8?q?hsOy2oD4TtSwC57iWao9pzBtEHIXD793P6J0MiPBaH8/0X5DwZlBIwE355L1rXWBQ?= =?utf-8?q?NAR34yDwYCHiOaJff29NEvN/S+A7VrvOkGGHvvUut+vjkdtr0M1HPx2ry9CubGsP6?= =?utf-8?q?XgUb4ca1ShLnF3YQ4tuYFS+4G/sU22Q9M1sEQ5bNjiTc3BDZtKLOeB8FFxcwgojA5?= =?utf-8?q?qJEjQG2QtIwMLtJy06uP+BJ0G+7k0qwoME/E6CYRqDsBgdRzQnGc/VeHDS0LFzPzH?= =?utf-8?q?kWmUZMCjkAXp5HPcMy2idZH8j7HIJfG8aJOPN5KJFF4wLAqmWEnn/rkGCJicBxbf9?= =?utf-8?q?XkKKN7zsoi3BkAi1s7IfjnBm1C+td2nCeOfucs7Jbr6p3TzwZWAF/9brs7QgrnVyZ?= =?utf-8?q?OEUiYfSWo6U?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:38.2415 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ab166165-3f6e-4c6a-da17-08dcdb0528b8 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E62.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6320 NVIDIA vGPU VFIO module requires interfaces from the core driver support to issue GSP RPCs, allocating FBMEM, create/destroy/reset vGPUs... Implement interfaces to expose the core driver functions to NVIDIA vGPU VFIO module. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 6 ++ drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild | 1 + drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 69 +++++++++++++++++++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 5 ++ include/drm/nvkm_vgpu_mgr_vfio.h | 24 +++++++ 5 files changed, 105 insertions(+) create mode 100644 drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c create mode 100644 include/drm/nvkm_vgpu_mgr_vfio.h diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index aaba6d9a88b4..5a856fa905f9 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -2,6 +2,8 @@ #ifndef __NVKM_VGPU_MGR_H__ #define __NVKM_VGPU_MGR_H__ +#include + #define NVIDIA_MAX_VGPUS 2 struct nvkm_vgpu_mgr { @@ -17,6 +19,9 @@ struct nvkm_vgpu_mgr { struct nvif_device_priv *dev_priv; u64 vmmu_segment_size; + + void *vfio_ops; + struct nvidia_vgpu_vfio_handle_data vfio_handle_data; }; bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); @@ -25,5 +30,6 @@ int nvkm_vgpu_mgr_init(struct nvkm_device *device); void nvkm_vgpu_mgr_fini(struct nvkm_device *device); void nvkm_vgpu_mgr_populate_gsp_vf_info(struct nvkm_device *device, void *info); +void nvkm_vgpu_mgr_init_vfio_ops(struct nvkm_vgpu_mgr *vgpu_mgr); #endif diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild index 244e967d4edc..a62c10cb1446 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/Kbuild @@ -1,2 +1,3 @@ # SPDX-License-Identifier: MIT nvkm-y += nvkm/vgpu_mgr/vgpu_mgr.o +nvkm-y += nvkm/vgpu_mgr/vfio.o diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c new file mode 100644 index 000000000000..e98c9e83ee60 --- /dev/null +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: MIT */ + +#include + +#include +#include + +static bool vgpu_mgr_is_enabled(void *handle) +{ + struct nvkm_device *device = handle; + + return nvkm_vgpu_mgr_is_enabled(device); +} + +static void get_handle(void *handle, + struct nvidia_vgpu_vfio_handle_data *data) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + if (vgpu_mgr->vfio_handle_data.priv) + memcpy(data, &vgpu_mgr->vfio_handle_data, sizeof(*data)); +} + +static void detach_handle(void *handle) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + vgpu_mgr->vfio_handle_data.priv = NULL; +} + +static int attach_handle(void *handle, + struct nvidia_vgpu_vfio_handle_data *data) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + if (vgpu_mgr->vfio_handle_data.priv) + return -EEXIST; + + memcpy(&vgpu_mgr->vfio_handle_data, data, sizeof(*data)); + return 0; +} + +struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { + .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, + .get_handle = get_handle, + .attach_handle = attach_handle, + .detach_handle = detach_handle, +}; + +/** + * nvkm_vgpu_mgr_init_vfio_ops - init the callbacks for VFIO + * @vgpu_mgr: the nvkm vGPU manager + */ +void nvkm_vgpu_mgr_init_vfio_ops(struct nvkm_vgpu_mgr *vgpu_mgr) +{ + vgpu_mgr->vfio_ops = &nvkm_vgpu_mgr_vfio_ops; +} + +struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + return vgpu_mgr->vfio_ops; +} +EXPORT_SYMBOL(nvkm_vgpu_mgr_get_vfio_ops); diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index d2ea5a07cbfc..caeb805cf1c3 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -1,4 +1,7 @@ /* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ #include #include #include @@ -132,6 +135,8 @@ int nvkm_vgpu_mgr_init(struct nvkm_device *device) if (ret) goto err_get_vmmu_seg_size; + nvkm_vgpu_mgr_init_vfio_ops(vgpu_mgr); + vgpu_mgr->enabled = true; pci_info(nvkm_to_pdev(device), "NVIDIA vGPU mananger support is enabled.\n"); diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h new file mode 100644 index 000000000000..09ecc3dc454f --- /dev/null +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#ifndef __NVKM_VGPU_MGR_VFIO_H__ +#define __NVKM_VGPU_MGR_VFIO_H__ + +struct nvidia_vgpu_vfio_handle_data { + void *priv; +}; + +struct nvkm_vgpu_mgr_vfio_ops { + bool (*vgpu_mgr_is_enabled)(void *handle); + void (*get_handle)(void *handle, + struct nvidia_vgpu_vfio_handle_data *data); + int (*attach_handle)(void *handle, + struct nvidia_vgpu_vfio_handle_data *data); + void (*detach_handle)(void *handle); +}; + +struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); + +#endif From patchwork Sun Sep 22 12:49:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809098 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2067.outbound.protection.outlook.com [40.107.237.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07AB2653 for ; Sun, 22 Sep 2024 12:50:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009445; cv=fail; b=ke+LEwAOjoTpK/Gb7UhzAc7dRFsbSgkMIrg5ohhKbjm6iqkkogw4nvw2300LjbRV9uBSGzoIMpYBdne+8dNrmv/MaiCbh35Yp2ZA+BRKz5r4lKCyPQSQoTx87GPrpdV5kKfmVcU+v/H6OwPlF5DK7XHJkfnI3Hu/mVtmrgiBr/o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009445; c=relaxed/simple; bh=0UkU5Ks/BhKyoiot64Tg+oNUo+iul+QHPPJdYV4LCKc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iPCJyQRMYueU7nEJi4GpQT6PnRksMhpBAz0ldBt33lDTeERYR7p2pyGTNFAI/GGzRNEF7SjBUHyT/d0Q+FaaRLBm7j+sL8fB6vtfj/uqNRxbBjNtkwMl+hsj+z3G2Tg11XlYihZVSiNG6KEDxcn6pVJfvaiAcfggQKt2SQUXiUg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=KgF4QpkT; arc=fail smtp.client-ip=40.107.237.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="KgF4QpkT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hclqAQJ14Hnl8Bl0HG7Y8RsgUIb9DYRy6y7CPKxGUA4O1aSM3vW3Mw9LeHdMwrtNYxJBxseanG2+6y3nafco1c0LD6an1Ed9j7AGlBhyorQ2EFXaJ4To6HXpUgkMKyjy1bcYFYbMzMQZLZcswpwLqG3b84vZJTUoO46Zz1RzbGvVB/tX1xAD/K5W/IO2ADQN5hV/A640Ha6FGfklLOvCsl4HuBIFicd9nsnF+LT5OJ6nqKDtoKQJbIRuIRFSdBIdnR3fhjHVq7Es7fYOuxrnE4i5BDXWbuEANQZi5SuoYq/rdlq17TmB6LZ5xTuont7X7xnNqNz5iPVuDzSBrZwV2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jqta6mOUHgbsTQz9ROpc3QRS94a2Cx+ZesDw9BiyDhE=; b=upul8Pi1hqzjVJND6RSj6X2vjVTWsG/C1SWWTmLrgyYRcXJz4VumkKAmuZEXcusfZT/J8F11XYmSwC4v/abv/7uXFYByaxEeOhw0ktE8eayUEOxqLZdFhg2MV18ojygNeqapY3QsdPaGPIJcvNkOOjmOitROjAh0HYvvDqYn0+HlTARfS/ZbXXSL3X1eaNH50SfKTYAbbbZdeA+CX12pgji8pBTXUJlFtIEwNqxHsvcYKst3iymq+Xj9ChReHZod3v5zTqnqCIIz1+3atD8TiUJ2Uh/FECbgjpdZg7QFXGyxfAsWF1JDSP94Kn52z8FyjI1rqPbQN/99p26UQePJFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jqta6mOUHgbsTQz9ROpc3QRS94a2Cx+ZesDw9BiyDhE=; b=KgF4QpkTMlEJ6fr3UglW4X5nwUyzHiQUydp4+jHJT1BppzEKoN6RZuVApLPeD30vJwFe9XeTYxaTILp9Q9c7Fzkp1RkXfZLFxe+88oaNpfHRmUh4ItywkrdhdOsWwpYxK/GpgHIPYpHXY/EXHpxgei2FDFB0Gd6IFdnBkoelIAzxfuBRsNIG1cSViMlBBkLSaYdZaSloX9gZzFCkprloYAnmScb6/83B38RYgV+8+WWAVmCc4gND41WU3QLV4EJ3M06qmD7wZLIvXOkO2OzoOHLazM3lfCR9LuVaO4VNvMn7DO12M/9qHQveB7UhCtOzAsVxGbZ2XeXNUDIXqitKmw== Received: from BYAPR02CA0060.namprd02.prod.outlook.com (2603:10b6:a03:54::37) by MW4PR12MB6873.namprd12.prod.outlook.com (2603:10b6:303:20c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23; Sun, 22 Sep 2024 12:50:37 +0000 Received: from CO1PEPF000075F4.namprd03.prod.outlook.com (2603:10b6:a03:54:cafe::c4) by BYAPR02CA0060.outlook.office365.com (2603:10b6:a03:54::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.29 via Frontend Transport; Sun, 22 Sep 2024 12:50:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F4.mail.protection.outlook.com (10.167.249.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:37 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:27 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:26 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:26 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 11/29] nvkm/vgpu: introduce GSP RM client alloc and free for vGPU Date: Sun, 22 Sep 2024 05:49:33 -0700 Message-ID: <20240922124951.1946072-12-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F4:EE_|MW4PR12MB6873:EE_ X-MS-Office365-Filtering-Correlation-Id: 2f53b844-1d71-40a4-0de5-08dcdb05284f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|1800799024; X-Microsoft-Antispam-Message-Info: vFe2JNNiKXW3aMDezVbTbF8JrkSZ2awQ73t3I0Q3N0XOmZXmW1uLERo1rGR5ToRCGcA8+3l8WcZOx9G8G7jPC0u1fRLTZDD1CI0G1dCDRT5NG/e5wmU+6eVF8n0sLWR2Ut1ad/U5Prms130HqAfW8DPn+ujSUCWB/7f3GbFboA7crOWMkskRuITz+4irs6zorFazFlGk5I2XE0NYaZk8FXfTA1vqEcwBJ24pxS4R1RinyrisCk3B6meXO2eQA3C16pReJIjGC+xBR+5ZNZTRY3+WZapAH03xUKNJKOGfcCnIwuRksKcqfChTKnDU8ids7HLsiw2HOJGvT9GgZd4s5Fyy8E5pSP9PsgUC4JUOg1KK6oCulCOKwEvxaVMsI1wchkLfhf8yfJgCbgw6BsNiBkj+3kTqLnBDH8MI0Ba1CDnZiQigDDF4quczCq37fbzjkchzr4ohikCXDgGn9dGN43+Z0Xt9udArqYzu52UwM2+EFX1+7Ld/gv+VdYAgbTC4UfFmN1l1kWwERUhuOZ4+8v4P8eJJvOvyZuhdYnsNWyuN4GLSjQQTcV4YSa4sZxfDm9XUDy5Hd6Ap/B/TYHZ/EsUpeQUta65kRhxjyGxhlgv1eVzFs3KRkxxdjZmhIwvLQPAeNz4hB/0KsKCqvcLBxEtQzG01NTfh3NDN+TMVONb/+KharbBIDuZJVGN5DKR6u94jL609a0O5RDx2XJSwXmfLMrQM4eNDxMBi52m2LFX0E9vlM9oyrthCoE753mURBXTW3WhMEYYk2WCtujaiHyLfsJecP7wbemlWE9S8xG7rxrzcKY6UUtuod4YMcqzvFo2YvAYNfCGHEvR5xXBQ7urfBZC3aR0GC2we3nLyvAJ3ROyZk7VjvqLGWptPthUXq9UVi/eRNhU2vg275QGUdlZ4LSl3AE3r3OovhuGCabDIs+C0Brb+TpAP55iUDX8EAhiMRkFCMdDmh/iPsClGoNavBExNsDPzhd5C0zQUsIk/jNfrGIj1C0VOxAMjwRS2gMz0+fo4PCKtcSLzQRfeTl7DINVhg2IkxHcUXb7j/Bw6wGb7BxsCPqM/AJ2XMhIjos0jc8W7E3zY1Mkg9NFHlfaC5p07E2mtYM09u3528f0QHRfxzSVMs+1ir7q3b0G2x/JFOPi6CbZm/9Sn1974/7Fj3gRqg+oPrhFEw6Mm4yjhZEVJGR/aNvuZwFEaL53r3EgCwFMN3kCVs7c8iB9ylcD25fxBYrP9GgPa8HKNs2NF599RZ3OAa4SV1ApTwJ9SReV8SnrjXR0pQk7Zp+VX6OrgK3Cb9O9KKNbhi60966SVZJ1oChXQl/kJpCAM9C5i/6WVVOnuilpGsrjua/YjoBF0fvcrmr9w9vqRRxzHgykgqNH0IovYWI7VCbiHhHfq X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:37.6419 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2f53b844-1d71-40a4-0de5-08dcdb05284f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6873 To talk to the GSP firmware, the first step is allocating a GSP RM client. NVDIA vGPU VFIO module requires a GSP RM client to obtain system information and create and configure vGPUs. Implement the GSP client allocation and free. Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 58 ++++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 10 ++++ 2 files changed, 68 insertions(+) diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index e98c9e83ee60..a0b4be2e1085 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: MIT */ #include +#include #include #include @@ -43,11 +44,68 @@ static int attach_handle(void *handle, return 0; } +static int alloc_gsp_client(void *handle, + struct nvidia_vgpu_gsp_client *client) +{ + struct nvkm_device *device = handle; + struct nvkm_gsp *gsp = device->gsp; + int ret = -ENOMEM; + + client->gsp_device = kzalloc(sizeof(struct nvkm_gsp_device), + GFP_KERNEL); + if (!client->gsp_device) + return ret; + + client->gsp_client = kzalloc(sizeof(struct nvkm_gsp_client), + GFP_KERNEL); + if (!client->gsp_client) + goto fail_alloc_client; + + ret = nvkm_gsp_client_device_ctor(gsp, client->gsp_client, + client->gsp_device); + if (ret) + goto fail_client_device_ctor; + + return 0; + +fail_client_device_ctor: + kfree(client->gsp_client); + client->gsp_client = NULL; + +fail_alloc_client: + kfree(client->gsp_device); + client->gsp_device = NULL; + + return ret; +} + +static void free_gsp_client(struct nvidia_vgpu_gsp_client *client) +{ + nvkm_gsp_device_dtor(client->gsp_device); + nvkm_gsp_client_dtor(client->gsp_client); + + kfree(client->gsp_device); + client->gsp_device = NULL; + + kfree(client->gsp_client); + client->gsp_client = NULL; +} + +static u32 get_gsp_client_handle(struct nvidia_vgpu_gsp_client *client) +{ + struct nvkm_gsp_client *c = client->gsp_client; + + return c->object.handle; +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, .attach_handle = attach_handle, .detach_handle = detach_handle, + .alloc_gsp_client = alloc_gsp_client, + .free_gsp_client = free_gsp_client, + .get_gsp_client_handle = get_gsp_client_handle, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 09ecc3dc454f..79920cc27055 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -10,6 +10,12 @@ struct nvidia_vgpu_vfio_handle_data { void *priv; }; +/* A combo of handles of RmClient and RmDevice */ +struct nvidia_vgpu_gsp_client { + void *gsp_client; + void *gsp_device; +}; + struct nvkm_vgpu_mgr_vfio_ops { bool (*vgpu_mgr_is_enabled)(void *handle); void (*get_handle)(void *handle, @@ -17,6 +23,10 @@ struct nvkm_vgpu_mgr_vfio_ops { int (*attach_handle)(void *handle, struct nvidia_vgpu_vfio_handle_data *data); void (*detach_handle)(void *handle); + int (*alloc_gsp_client)(void *handle, + struct nvidia_vgpu_gsp_client *client); + void (*free_gsp_client)(struct nvidia_vgpu_gsp_client *client); + u32 (*get_gsp_client_handle)(struct nvidia_vgpu_gsp_client *client); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809101 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2075.outbound.protection.outlook.com [40.107.92.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E722F2772A for ; Sun, 22 Sep 2024 12:50:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009448; cv=fail; b=O0zCdWlk7oynFXdBvfb3stnRfNwQTk4RfL6qD6VLKN9dqO1nUL5Id3vDnYHCK7zEBFcyFKbZI9QQhZ3nZVsl6FmE7F2HicNb8XztQf/5G/hcAImRdLGbcMCx3g9DSShQaiJMSBwS7WW8quXmIOXX0cbdKdDkAt4ZrEduizrfohw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009448; c=relaxed/simple; bh=m9fQylcUQHiyhyavuZGnHNm902zHZQcACBgfCF1RmZU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Nr88FCMybJ4S+Zy3mc7FHmcQt7jkEWR6AwV8AbEAFUqT1TXbnHWiVq3QxZMwaRW3x2n7vq3ef6Ol8aHdWRYW6ey/cisPQszhFM1P6BkikJpH+G6eVKPmKoPz9J9qaeJ+Fl0IgDkxFaY07JDj3jnrzlRcw5P39GHMlJYhmo4d0A4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=mX2CO8vc; arc=fail smtp.client-ip=40.107.92.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="mX2CO8vc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hS0qOQzu6jBkjB62/70hZURsyJkY/WDnlMC9afTufdMBc9/ZyQ5+w96XLHSfHwhD52Le4ZJg/lqJ9eqfW4Z6S54NGAHtr7nbsAg//UQR8T5q34enC7V2R8VQugv+NIvyq0spcONBdhBNZMRQv97DcJuqceTTXon4tp64qzYHHwg5gwlvZ9tud+nJx3dBoVKmDzy4EvAdfgogaOIE1risG0x5wMey/unqxQSmiM/Ww4jslPaL146GlGyBO1VssRwheXS26Apvf5HX6Tu1WXopmoS8zIxkqTHfJ0uKtG0rmH6hz7s88KcADk4c93RWMxqzW15XnjlJMTiHJcAy1um/uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=x5HwsorBozNLwHCTmg1xs5zMeJsUudr73qdpO0DY4xY=; b=gE4NSY+8NGMobnJtQhtQ9USaAqXuqLLgAdmBQctm2t6/+YSg2MF2QCwmXjVpSpqjsWmvunx7DgdioEIqW8pLKey6vxL7rz5Ky13A8HTEV5Efusk9vVzcrkRZrixS47z3rxb72KPYOYvyv0OvlPRWLEuwExTY4DjLhDix/GQOBqFG3Q27ndIf7I9hFaC1ZVB51ZgZmlkk8oJk8828Iqg5hEN3JwZ84XqHQ2uUCmXW9tTav9WbV7qaWI8pug8v0KQSCa+eony/QBLMu2tT+vlaOhHSgNRU+iv7RX8VIDtfojZKZxrdvKcCOMhIePQ+td10g9X7KEvZL55xUglAyVU4hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x5HwsorBozNLwHCTmg1xs5zMeJsUudr73qdpO0DY4xY=; b=mX2CO8vcacNG5snVGyyymLFsmGHINbz23rUJc85q6wmgqrGMcClGxyDZAKB02wkOOJK6S/oPst01CJDawrsnRNxs/ovfX4uuzmeLJUCL53QgpA/f2PXwjOMrkVn6kg/oyIgNM8kzdan2gNG0C1YjCQgScfdXe6gd+qouuOw9aGB7JHKARRoQAUHiRZkMpV+HldfY70ggNxPEyM0RR9Q8tR02cnQyO5cLg0zIi6smb4jgKVA/1MnfAk1/NEfp4mkCqyWqqmDguxML9Lv7uz8qeWcBToEmdaMqfmm3zdbt/Xg/Dt/OSzOnNZTfS9nUhBLfdj5aE6j7NTxNV8ufYSlh2w== Received: from BN8PR04CA0045.namprd04.prod.outlook.com (2603:10b6:408:d4::19) by DS0PR12MB6557.namprd12.prod.outlook.com (2603:10b6:8:d3::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.22; Sun, 22 Sep 2024 12:50:39 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::3a) by BN8PR04CA0045.outlook.office365.com (2603:10b6:408:d4::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.28 via Frontend Transport; Sun, 22 Sep 2024 12:50:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:39 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:27 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:27 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:27 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 12/29] nvkm/vgpu: introduce GSP RM control interface for vGPU Date: Sun, 22 Sep 2024 05:49:34 -0700 Message-ID: <20240922124951.1946072-13-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|DS0PR12MB6557:EE_ X-MS-Office365-Filtering-Correlation-Id: 44283285-a986-4122-cdae-08dcdb052957 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|1800799024; X-Microsoft-Antispam-Message-Info: cRFZ9Jyf2MaZCKQS+rUTIhUcfuERrZY6HsVThmGIdZp10AQiTivx80bdlFWfx3Ga3L6HXdHQFX50H/FCjeKPGtWrACDr1sn0pQbVo4LH6I/f0NDIml3oN06q7aYFfVbMQlNzuytDUXRwmF//xcQCzb+knv1BvhycKWlt5Vp5U4vQl81AULEQF5AGyqQpz25qzOFquez0vyjULFTaC0HPvIelURithh877wcCNiYI1UxPeCgct2nnmqJH/eU+WDkZ1XPwakFge9FywFwT54IZmFkSMUZ4BEfMK71esrbV0QOUrBAWB3JOYkwajwuDimOU1JwlEmEWzYcCAGv1AC1FPV9Pnqln4tiYwLDEWOpy1qDsThDbiN69Ou7Q9R9BiZFUs/HYfB6oQH+adWpC5NtNiuG9oyu81Hh+IejWBP8H1dFC+81zJOIKShmOPrRH6967413DBd2GrXMvea9jlfxhs6mjffOxtV2o5UytqaZRKMJitaNgZlF+A3ynZ0oPIOevnLNHD6sgNXekQutJ0auWqVZHWK+3T25gWL5g5j69SJqagVMcr/c8pRabQaDmiKmu4CdMBWE9FjEho/0xeFXRcw1uJ8G/BnAuEfCx8KRoTxLUVML8KdTqWK+vWofxIEjbh5KxwTfk/u+3axkGYvgA+TaUFgBcPxicUjxXSr/YxU2uLgb/K28edTqO69ek8Lrp312Ckt+gStHvl2hkf6zYtgxkUOgS1kL577fvWe9caZk8KYybkI2n+DUJ1bcHDvziqq0l6enQYrVlAbchum57KiQoV/KWtrbjJ0oZSpm7y+5bMr1W38M9+5x8sg5YXwOYJmgE5KNzaJfs/NmLK1/Do384YwUpC3NcKldVZfv0Ngqn4c16oivbkgvT3Wn1Db86Kj19EeIYXOg246OJi4v6s7YYx6pFy3tmVeKtnNJ0V739nIMP3KkKHkAsKvvy6VupReEt8GK7McnE6O4uM7xTpjkYz7eK7C8GH5LV8zb1pdXbNvmiPizYHDHc4n96MxAmQwcU51ON8wG1PDsk2f3hx/OhVBQ85ah+c3vINzgyywm4ba03HbmkbjhL6bnas8zuTCEmpodcRogEXRYD089Ka4e9B2niGmO7tQSp7BVs7+lmwuN4IuY+R7HbhVR5X7clxXeU/ekmL2K0vDtBf1gnmfxJvFivyKr3DQG8HgujTYYVjM8COoDCmjF+59iCNHUsjPngXKHiX8KnAJT7BPJws7MimtH//HptZE7S4eQ8MxXCwa/TFWRIcVjW2q28xZY9uS6ueYsx2g3U5zwCIWGx18Y9+Qo+hqNZ2LLH8GbC/PT+a+/En/yDNjzqGzyHJPC25uZ0xnM0RwqLlGBTApvk7V1ShomEIMh3cI0G90ErMgzJPLk5kDSLH2emHmMTkUqPImCloPO/3kx/PkSyk3F4UcDXjgmjuVRBnVAZvWWw1nWpyFwqQ0kzUfiCxEGQP4a4 X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:39.2832 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 44283285-a986-4122-cdae-08dcdb052957 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6557 To talk to the GSP firmware, the first step is allocating a GSP RM client. The second step is issuing GSP RM controls to access the functions provided by GSP firmware. The NVIDIA vGPU VFIO module requires a GSP RM control interface to obtain the system information, create and configure vGPUs. Implement the GSP RM control interface based on nvkm routines. Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 34 ++++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 8 +++++ 2 files changed, 42 insertions(+) diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index a0b4be2e1085..9732e43a5d6b 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -98,6 +98,36 @@ static u32 get_gsp_client_handle(struct nvidia_vgpu_gsp_client *client) return c->object.handle; } +static void *rm_ctrl_get(struct nvidia_vgpu_gsp_client *client, u32 cmd, + u32 size) +{ + struct nvkm_gsp_device *device = client->gsp_device; + + return nvkm_gsp_rm_ctrl_get(&device->subdevice, cmd, size); +} + +static int rm_ctrl_wr(struct nvidia_vgpu_gsp_client *client, void *ctrl) +{ + struct nvkm_gsp_device *device = client->gsp_device; + + return nvkm_gsp_rm_ctrl_wr(&device->subdevice, ctrl); +} + +static void *rm_ctrl_rd(struct nvidia_vgpu_gsp_client *client, u32 cmd, + u32 size) +{ + struct nvkm_gsp_device *device = client->gsp_device; + + return nvkm_gsp_rm_ctrl_rd(&device->subdevice, cmd, size); +} + +static void rm_ctrl_done(struct nvidia_vgpu_gsp_client *client, void *ctrl) +{ + struct nvkm_gsp_device *device = client->gsp_device; + + nvkm_gsp_rm_ctrl_done(&device->subdevice, ctrl); +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -106,6 +136,10 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .alloc_gsp_client = alloc_gsp_client, .free_gsp_client = free_gsp_client, .get_gsp_client_handle = get_gsp_client_handle, + .rm_ctrl_get = rm_ctrl_get, + .rm_ctrl_wr = rm_ctrl_wr, + .rm_ctrl_rd = rm_ctrl_rd, + .rm_ctrl_done = rm_ctrl_done, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 79920cc27055..29ff9b39d0b2 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -27,6 +27,14 @@ struct nvkm_vgpu_mgr_vfio_ops { struct nvidia_vgpu_gsp_client *client); void (*free_gsp_client)(struct nvidia_vgpu_gsp_client *client); u32 (*get_gsp_client_handle)(struct nvidia_vgpu_gsp_client *client); + void *(*rm_ctrl_get)(struct nvidia_vgpu_gsp_client *client, + u32 cmd, u32 size); + int (*rm_ctrl_wr)(struct nvidia_vgpu_gsp_client *client, + void *ctrl); + void *(*rm_ctrl_rd)(struct nvidia_vgpu_gsp_client *client, u32 cmd, + u32 size); + void (*rm_ctrl_done)(struct nvidia_vgpu_gsp_client *client, + void *ctrl); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809102 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9942B29CFE for ; Sun, 22 Sep 2024 12:50:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009449; cv=fail; b=qnkld6b3dqGWHHMjDFzL+2JMfm9zBDKWvcbRwI+7NzT8yEUugPYQQjL5PcNIpDaiE9K5sRC00fgGneEPJQzQa+nlTTDxyX/KV5l47DCOfFPkvq8BpsRUl7Y68DtTcOCclcDRhXzWAMKaNhSwb1u1N4SMOGc7FIkgJmlWrbLSvVE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009449; c=relaxed/simple; bh=0maYBK1+RdEvZo/xKDMBqteT9pl26EhIsUjVzfHXR38=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ozP0sGjdMq8RBdrmathDHnNX2fb7QYkxk6s4P454pBhS0n0WaFe5mc2kAPTnPVbYQ+U41BXi4Hjahg8S+2d6VNYDGqqpv+VXJUqWbU6r+Ay0ljEWOWq2Xg0IhMRb6YjZVZwodiA+m45N7enN3IA/xEoWfOomdypW53m6fka5fV8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Toecuze1; arc=fail smtp.client-ip=40.107.237.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Toecuze1" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yvljXu9tt4sT8dQLp1YM0Lc0tJqRfOk79tLL70a5FzBccCbAEveK8YsUXS//l0+H4/pyD6hlo/qTKq8NP+YM9iOopUNubCFZa6dSn3e6lwjRU3LNsOcfLSm+0a4GNzQythlnlMjQXWMPmX7vO/PF0pM4YaKnGhySMJ+i/bhrNFYvQGX9kkx/ctdR/bbkebAu6kPWAonXqC7DvwRFx3Tc6y/2mmnxN2janJ0Z/gewV50Lkq57RPo6RvIdf4INPJdVCKoYmk3TWunlss6sd1wl0FE9TwIjVRe8aBsZTC4A3GJkBnNaG20Zbvuo1vUvtYABq14U84JGu1JMR+/eBJrnmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qXy8Q8t9lms2cFPdSfKW+hYTXiuejIlAVtlqQ8q7Gok=; b=yTLnw0W+KeX/YTMkMPCiExj05M+ncjEPUkXt9JqBoMYwI/WIs4Zkw3HIrN7hu2Pc0yltZzreEra1L0njSK7Wt+Yq2lE0GCdyQZAUKzS6XTm+GlTiyNjaw6sJ51w9u3e3Eg20WMXay0MIcF5/eEE00r43RHzVMAb6DC+xlGEV1a27sLbsKrOU93ja3X9jS4l7ofKxSKkYgzdKa+1gnLuiXU+WgLzssGD6AWev4/qpqL6T02+70QJ0il749bDsUurQEwrMr4+knGiYTMexMPS+AR/IgB5ys6cYgf9tOQox+wvF36lgUe4MOE0TWGNAfLwfwdcnE24CWkA7oWaxI3Ci4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qXy8Q8t9lms2cFPdSfKW+hYTXiuejIlAVtlqQ8q7Gok=; b=Toecuze1nwCaFF/zFGArmoDgvTjPnVWRlXOY/mFHFhnSxdrA+UbvU6MHLbrGfA/0st21L/JDFvmuOQtZS6T+UhEyECzO/TIuHriyKHC4aRd6hbCHRRiLYE2xIj46DVktWKz02UdGmTADSyopnVTRwTiN7GZsBF6I18F8d0QLtcKE8h5QIUqvmm4ISrvCDLUc/ZW6bl8qtbdFkbe5NbxM6UvEkC3X16dcdrisO1CW+7NDt4Z03iKIoeCttSOcukSJN2FWegaGlud8xD5+H0YgtITy7Mmyc+FIaFJrFKOlSLQ7AD7OLa4TIIaa29p13nUydVqz91mXN28mTZx+iYNfWQ== Received: from BN9PR03CA0286.namprd03.prod.outlook.com (2603:10b6:408:f5::21) by SJ0PR12MB5609.namprd12.prod.outlook.com (2603:10b6:a03:42c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:42 +0000 Received: from BL6PEPF00020E66.namprd04.prod.outlook.com (2603:10b6:408:f5:cafe::29) by BN9PR03CA0286.outlook.office365.com (2603:10b6:408:f5::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E66.mail.protection.outlook.com (10.167.249.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:41 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:28 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:28 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:27 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 13/29] nvkm: move chid.h to nvkm/engine. Date: Sun, 22 Sep 2024 05:49:35 -0700 Message-ID: <20240922124951.1946072-14-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E66:EE_|SJ0PR12MB5609:EE_ X-MS-Office365-Filtering-Correlation-Id: 0f01aaa1-c603-4710-7b59-08dcdb052acb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: u7hFjxCpjtEJZiBAJh11OegKK6LhG6Y95uafhxeueKKl2J8pBa1+ReYT4FYyfFdQny4zHaFvlGKE3fUL2wVC+jDJCfiJrclrczZ4jGexDQ6tpduh4gmylPjco7TiJx/X7OZR+YAZE5FCiuezuBHVnX07NUNbANxKo1t6c1stGg6ecNROCWK0t8J10VpDJAs3Ns5zkSPDCIWAav5I2W/7IFAVbMiTVkms+rGAZQBWdlljb8DY4Th02JA1neL/5+OAatc/k2uG2LQ8ZxFJWqXscLtJfsOpGtiQyL1JNv/GbcanKTie9fKPBD/TlR3X67Y4w+0ZT80hwPUlJmb+ht7hzZOwHZCLHZd6+7FWiS4Hkr31dPyNoFd0FrbG5X6NyN3c8bvmMzjsYrYRJWHjhJRCeRGSME/P2vP/NFe5wa8M5W0jQXx7pnzqR06aWLYCxKYHXs+RHRMUPs+GljMZioUxBuV4AXtO72XYMKHIsaP+WuZJxLZtrK4pcrMD82XVFEwOneGUUAcSlrdFkjw2fjfx3b+1h5JpJ/7DTCQxshktg6nBh7h98XucqYw7zPoguLF2X1RPo6NoHmSPNpetPr69VzHItsyNyewhHGPbMN+JNzjSJ4nqbU8NsGdjDJjzik2hIQqPWkf+ijq/NqNyiPi9aLnQrmuUYtIashQhlDuDUeKUUMj9XAYyRljvAFAQmdSnb0bto2SJXJ8gTahMuKgOqvdMFB6xm15PaeHwmOIQZPM5hR6wYvBKE+6rHUNhRq4whNgROUrtglStK4iAKbzmMQhXAfGr87l/MoaBGSV9FKXoWSOJUsrC9E/3fKFZdreQKidUoQNa/17IK74yJ4kO8gYE4qYI27EmLkZjXnPY/FeiQ85pHUiNvH6IJMTEd33TUBB5IPQyFkn143Dy6HuvR6V1MnqSMW7l6xqEDBBKYYIqCf257v+9WmjBQ5zt+N/MrOPJ1AanHpxzA81h19uiKTQAUUoA07xwsvSr1FurtFMo0QACFLps+Q8fzoV4HbC491kM0MFqM0VPLORWjfEtf1HavgQ7oXt2QB1KcoBPVJ/v77zvM70uZS0bH+98pegVenO6Njd9iZzDpT+/2Qkwr/IMvZA2YgQ7zeHujYv5wwT0Xl2425pkcG1KumUwkL5KBYd4CnDYq9rxLZGPH0RByo0ctQjsk8vFiXC7mtk+/gbhgNbmYENmwKweo0H8Su7pam6oy3Pv7DhxUWOrhghygseawPsRz+BaWHxiEtvxGwm9RWfM8aUZhZVZ+/b5Dl2+BbFWUSiipXUPUux06MUWGuI9XxthpWRgkmTz1ipxKoGGJ7GMVrfUZ9bfLPeWryOz6sShwtJ/WuB9IPDxrrm+rfc6qdiQGP/hHWMtojroH9A= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:41.7190 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0f01aaa1-c603-4710-7b59-08dcdb052acb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E66.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5609 Move the chid.h to nvkm/engine so that vGPU manager support can expose the routines of allocating CHIDs from the reserved CHID pool to NVIDIA VFIO module when creating a vGPU. No function change is intended. Signed-off-by: Zhi Wang --- .../drm/nouveau/include/nvkm/engine/chid.h | 29 ++++++++++++++++++ .../gpu/drm/nouveau/nvkm/engine/fifo/chid.h | 30 +------------------ 2 files changed, 30 insertions(+), 29 deletions(-) create mode 100644 drivers/gpu/drm/nouveau/include/nvkm/engine/chid.h diff --git a/drivers/gpu/drm/nouveau/include/nvkm/engine/chid.h b/drivers/gpu/drm/nouveau/include/nvkm/engine/chid.h new file mode 100644 index 000000000000..b9e507af6725 --- /dev/null +++ b/drivers/gpu/drm/nouveau/include/nvkm/engine/chid.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef __NVKM_CHID_H__ +#define __NVKM_CHID_H__ +#include + +struct nvkm_chid { + struct kref kref; + int nr; + u32 mask; + + struct nvkm_event event; + + void **data; + + spinlock_t lock; + unsigned long *reserved; + unsigned long used[]; +}; + +int nvkm_chid_new(const struct nvkm_event_func *, struct nvkm_subdev *, + int nr, int first, int count, struct nvkm_chid **pchid); +struct nvkm_chid *nvkm_chid_ref(struct nvkm_chid *); +void nvkm_chid_unref(struct nvkm_chid **); +int nvkm_chid_get(struct nvkm_chid *, void *data); +void nvkm_chid_put(struct nvkm_chid *, int id, spinlock_t *data_lock); +int nvkm_chid_reserved_alloc(struct nvkm_chid *chid, int count); +void nvkm_chid_reserved_free(struct nvkm_chid *chid, int first, int count); +void nvkm_chid_reserve(struct nvkm_chid *chid, int first, int count); +#endif diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h index b9e507af6725..a9c3e7143165 100644 --- a/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h +++ b/drivers/gpu/drm/nouveau/nvkm/engine/fifo/chid.h @@ -1,29 +1 @@ -/* SPDX-License-Identifier: MIT */ -#ifndef __NVKM_CHID_H__ -#define __NVKM_CHID_H__ -#include - -struct nvkm_chid { - struct kref kref; - int nr; - u32 mask; - - struct nvkm_event event; - - void **data; - - spinlock_t lock; - unsigned long *reserved; - unsigned long used[]; -}; - -int nvkm_chid_new(const struct nvkm_event_func *, struct nvkm_subdev *, - int nr, int first, int count, struct nvkm_chid **pchid); -struct nvkm_chid *nvkm_chid_ref(struct nvkm_chid *); -void nvkm_chid_unref(struct nvkm_chid **); -int nvkm_chid_get(struct nvkm_chid *, void *data); -void nvkm_chid_put(struct nvkm_chid *, int id, spinlock_t *data_lock); -int nvkm_chid_reserved_alloc(struct nvkm_chid *chid, int count); -void nvkm_chid_reserved_free(struct nvkm_chid *chid, int first, int count); -void nvkm_chid_reserve(struct nvkm_chid *chid, int first, int count); -#endif +#include From patchwork Sun Sep 22 12:49:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809100 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2078.outbound.protection.outlook.com [40.107.236.78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D4551CFB9 for ; Sun, 22 Sep 2024 12:50:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.78 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009446; cv=fail; b=CL/ik5pHBEavdwq/fynlQzdsOHM5vria7gk/ozqmVxnpZ73pkOJrhAp+/cgcsRCBKjatUs459SPGhF6xE4vR9otMWu4Vv94jj4ztAbNFQ3V1skG9jMfwTejty4jWIhZsWumXgHyXqNUu5AZHUZMLuTcYKnAJHXiI+eGHaBPt3UY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009446; c=relaxed/simple; bh=ubXn/ucNtgjj5IoipzCCYpImjOJ/yOpTBVUichnzssE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=s7roNN5YPqcct5grV+GeeQo4GGRNOE+efoh8x/468+GzxGJ0+ldijCh+O90sTDuNAQKS+YDpAGLEQqAnzIngEaE70rCYM4JjSPFckfS0/+xRMOoXl+LOajttMEST45T7ZQjZ6IUgN/d2+tgdJm3PydxLbtTFEKv4r4eo8+nptDQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GWU2osbL; arc=fail smtp.client-ip=40.107.236.78 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GWU2osbL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=c8C/l02d/edE1675dx/0nbBSEudUqYjkZ0x5lo5ywHVUBl2KiyNSUSN8KrJcXeiwcs/74b0en6eZ7FDlfipiwdfDHDYXWJy1biRWRoIo3/429DSWhjS1I0xjZc8Zsdw3dbCPuN0PYl4saW+5eTa/FyDQx9uedL1FCdqgvMV7vuzD7Vk6/ilLjhhhZQmxP+sVFszBa7RSODb6hxr9T7D7kBoxOxeIEdTNOudePGdEB+iGyCqmJLDmcrCZMbHQTgaLhhiDtrdC/NxZDcQ+1AYgMqhtaG8COP9jo5OKjiTLjsW+cDM61efWCrHT9apN8jPK0/fg87cCE+scjR+S2kZcHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BFW0UGbSbEJT0csX5FRaUj4dkydgQ376dRX2pBfaXq8=; b=DO+4I/gpQQiQFd0BQRRA2vrBJaZwvY3GfxwA/vsbt/Ex3SCwQ3VJ5yIueDoFZCDj9kErSLoO70nYVmO6NIijVOvqpku7fCGr6QcbKIKXAbOdV3YodYGc7y0TGt06drkYtwmG65I52WlI3jOe0WOICZ5kbIp236oiJtTJmAC804qJEkAXJA56Y2btZD/P0XREBU8GLH3VFCrbHoBQeRoU/YcV/1A4qPa5VN+eyqUPnU6WN1+wgekOcsVnJprToA4USSSPCxltiM0e2MuPWK90aOMIZUSRzTEmDf7QVxPFXVR0fRkyn30gSJPr9SV8ebEZYd5Pc6icjr+P+nTyosIv3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BFW0UGbSbEJT0csX5FRaUj4dkydgQ376dRX2pBfaXq8=; b=GWU2osbLA1wA1GocdAq8hG2CQGvdyxyHWMuq1kzPblsg8qkHJgnQx0tFFJ1FEGE06q05XfaJ6xtCvMwDVsBXG9cgCR4HsupNxFYTnf55+u6mMTUcEaox2mJDcW9YVMBU8mIJ0AzZtYGrZ+Bl4jrhCH8JYBjtDsWd9CUiYdMpVSs8KjKdyr2TSdXykzPwlPzeOSv/XuL9zwQZA/g3MSFFTSUDmK1qIqGizDAl+jjjIsiZVM1JcmlGk5eA8zByntzchHR8XHQ9/3HxrNCf00OTCjIM7EbhYZ9lWc7AEEO9BUaPZnJVBukhZ5sHK/d7S6JwRQBOUmLoHJxh1/8hvtuRDQ== Received: from MW4PR04CA0308.namprd04.prod.outlook.com (2603:10b6:303:82::13) by MW4PR12MB5665.namprd12.prod.outlook.com (2603:10b6:303:187::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.21; Sun, 22 Sep 2024 12:50:40 +0000 Received: from CO1PEPF000075ED.namprd03.prod.outlook.com (2603:10b6:303:82:cafe::26) by MW4PR04CA0308.outlook.office365.com (2603:10b6:303:82::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Sun, 22 Sep 2024 12:50:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075ED.mail.protection.outlook.com (10.167.249.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:40 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:28 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:28 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 14/29] nvkm/vgpu: introduce channel allocation for vGPU Date: Sun, 22 Sep 2024 05:49:36 -0700 Message-ID: <20240922124951.1946072-15-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075ED:EE_|MW4PR12MB5665:EE_ X-MS-Office365-Filtering-Correlation-Id: ed54e816-3140-4e03-9074-08dcdb0529c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: WOyc8v8cfO6hJbFxtDuilgs7Nazy4xoGogiIAlLT27IZYD8FRTuN8tOkIR/H/XoMP/lRAoiiqAjUq56H3FZCYvvUw5MtxgT+0+MkDhcA3ko0DXc+4PvnDbbUJw8YSNA7avGmap86nSI8atd24YVcl9i43LsD1Ma9EhLhWtAARF7YJKCLZmq2abESBujM+CUnWVhvI5FQtxxmLu8Sr2CTHwSThJbAXeUuUcpsfR1LUAHpjE6ETYdUmMBa2pwE1fCCPhPQPZ+kN271zQ0k4/3QVNy1lcNe0uumaXaWmYpxUtnsRseygJZAnCcfihQxQ/XiNsIcwjY0PQQLpWrDnYX28Dm24zvAyn5sV4Rk7KLqW0MCQnuBFRVtIRzMLLGpDbimG9Zudj0FQRggMDY48QvyR9CAV97X6iCl8weX7qExzsQXEgebmXnOb/aIc1+bbGX/1qeYGvLIYtF9bfkIf2eS0Vrm031/nyUtahJw1C3oM3RRx7izO/PkuA5EI/i6YP9iBsti8YGfy6V3v6Pt5o9iy7H4NGXRM7vX6UV4C0IS3/fywmnVZu56ugfOLscixM6Kt+39zufmkdyBBGobNcN1yOjuyx+o8g5cBaH+HUDc0BDU2NTXXmUQbm279kG2dvir8papONITqNnzDSI/4Zlm9FJqL357tYpAF0Iy0FeuhISdNilJEiVYVtaBcjyJk0eoXJy665eAfSPfLnWKP4ACo86vjM2gRLam61qppnXnaVK6pnZKOy+GHNzjfaCMb9mI2+pSNmc0Y9ALBG4KjV/d95Ot1Cx3sJuXrXhKeKx1iaRxhODRGYdq64wZk2zE/vIOSlnUMAHK+yIiYN3Kpk2sFQSG+slcU/5bIUSH3bfUOzsCuQgiux1UNmimAtDZ+ju6ojAoVYI6pUgSmHTKvL7juxwPptH/aICYql1lEbjcKoOT3Ge3OC+48sbO6Mn2KGhcys+xc0JZkbA0+cblhDiEw/XIcQfmc8guXovQB3LVQquDAnySp/mmvxBIXXfdZM+FnNmJLe+kPbngLu8G52w7Lu7iVQkUdnD5Xs98elBrWxV1CdQubdPXjZz1ZK0qLgmSr9fT8GoCejHFdZvHeans52QjggvTcycsvSmPCVCokhEccLrF8oktAkaqIfAfAT7oX7CsfUz53K10WJpJa0AuNEuxRAxZHOOZFk5NUOsySqaQmqo0ZVNsY90NnzgH4v/BA4uNwok1sIDiXirxjDTEYIziP1tg2XK1L//D5w+sL7pZT5HrGUh+bkRwwrefIKhCPMKBPmi1ZPk267b0U9UrgYtlLcBQ5zqi8vI2QvuvpJ8ahsI3Vb8dHyns7hLegeYg7kc2368Lye0Te+GDHnRAhvBui7GYxB5wRvwltoFSQOW9HUxueGkHncPt11+epsVO X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:40.1053 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ed54e816-3140-4e03-9074-08dcdb0529c4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075ED.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB5665 Creating a vGPU requires allocating a portion of the CHIDs from the reserved channel pool. Expose the routine of allocating the channels from the reserved channel pool to NVIDIA vGPU VFIO module for creating a vGPU. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 2 ++ drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 28 +++++++++++++++++++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 2 ++ include/drm/nvkm_vgpu_mgr_vfio.h | 2 ++ 4 files changed, 34 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index 5a856fa905f9..a351e8bfc772 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -22,6 +22,8 @@ struct nvkm_vgpu_mgr { void *vfio_ops; struct nvidia_vgpu_vfio_handle_data vfio_handle_data; + + struct mutex chid_alloc_lock; }; bool nvkm_vgpu_mgr_is_supported(struct nvkm_device *device); diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index 9732e43a5d6b..44d901a0474d 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -1,6 +1,9 @@ /* SPDX-License-Identifier: MIT */ #include +#include +#include +#include #include #include @@ -128,6 +131,29 @@ static void rm_ctrl_done(struct nvidia_vgpu_gsp_client *client, void *ctrl) nvkm_gsp_rm_ctrl_done(&device->subdevice, ctrl); } +static void free_chids(void *handle, int offset, int count) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + + mutex_lock(&vgpu_mgr->chid_alloc_lock); + nvkm_chid_reserved_free(device->fifo->chid, offset, count); + mutex_unlock(&vgpu_mgr->chid_alloc_lock); +} + +static int alloc_chids(void *handle, int count) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + int ret; + + mutex_lock(&vgpu_mgr->chid_alloc_lock); + ret = nvkm_chid_reserved_alloc(device->fifo->chid, count); + mutex_unlock(&vgpu_mgr->chid_alloc_lock); + + return ret; +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -140,6 +166,8 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .rm_ctrl_wr = rm_ctrl_wr, .rm_ctrl_rd = rm_ctrl_rd, .rm_ctrl_done = rm_ctrl_done, + .alloc_chids = alloc_chids, + .free_chids = free_chids, }; /** diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index caeb805cf1c3..3654bd43b68a 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -127,6 +127,8 @@ int nvkm_vgpu_mgr_init(struct nvkm_device *device) vgpu_mgr->nvkm_dev = device; + mutex_init(&vgpu_mgr->chid_alloc_lock); + ret = attach_nvkm(vgpu_mgr); if (ret) return ret; diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 29ff9b39d0b2..001306fb0b5b 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -35,6 +35,8 @@ struct nvkm_vgpu_mgr_vfio_ops { u32 size); void (*rm_ctrl_done)(struct nvidia_vgpu_gsp_client *client, void *ctrl); + int (*alloc_chids)(void *handle, int count); + void (*free_chids)(void *handle, int offset, int count); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809103 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2081.outbound.protection.outlook.com [40.107.95.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812931803A for ; Sun, 22 Sep 2024 12:50:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009450; cv=fail; b=f0ZJQkJaySNwlRoTlJf0TqKlbmJvbVS4kWqIIjWPiIsfXQ74bB9NmlGPZmWd9NujHcn5+TQqXqpzjBQLSpR2Xk9G3aatdIH222CmynpvA+Xh0GBzEPclg8Pgw9xKnoreb9AF7k4lC8l3yYFfmSA9ofcsirJBz6CFkUoktql7E1k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009450; c=relaxed/simple; bh=mvaErpb73mjTNY/zd1F0CvK3v2ZX4b/VVqTY/1RnjF8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Q2IgYkhbW+yAOXLZPpq5Pymd5IlLO+8B26zEY80kv+rd/M67IhOfyJOvOZxSsNGs+57kvFcKW25wRVT8NxFxZk5Ubv5t5e+/9h1VS6YuYcVSTCUXdfZBo4GY9XfpsSP6fhRBzJECgtIu6uSM810dhQ1SkcOcziwumLTPzl2vjUY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=DI61n69/; arc=fail smtp.client-ip=40.107.95.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="DI61n69/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EJtJ3oQwuCXyFZ90UWpUXxN+Q46L4nsU7egHncUSr+DVXrMMOaB3foR1rWJrQLBoU8jBLcVleQ9/Lu6/kHcFLwBOswajZTCFcdAD9O4LeEU1YQqvDkRdI2nPNIi/piaNaFBTLpoZbL1Ijqu0ZyAzpLopcJDuU9oBsOAXSM1ZpUQHfr/ehL8e+rC2GRHHg+fDaSixvwOcgyiKzZmD8+f7zizQLjVtsAi/S1wK8Rfhzd0XtM7oQ7uUrCEvGGPws3dfAjSDpbeSVQu12moD+eTB/xCTKpgQHgTAKJrs9mYj3QrP/j/bYSkmYTL0oTHJJ/WWA3vm+VnI0i5bvuGrsaCtyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NYE+Aay3FUrAyLLloByMaZmI4Qgq2DLngBI4ghqmorM=; b=H88FGj0oZiWuCfMhIayNWG5nDxYq+6TkQQXAW3WuweVwpHi0S3NVY9+nr18joo4DwotdosSGJmrDJ0gUGBcjGAom8zaBCHyDmWd9lBJy+Sk/elh9aieWnL0fmdLsIgS+MKkRkGYTHPmmjvYwd2FKNiNotf5xvWCRk5zi02OcwOW07gpTso1QeqZuSLqAVfBHgVsRGcTQti9xlp4Pl6hmeiqLCGmDooCHQUpgJgLQaLRtWdluti4wP7zE0k/p5OQlUVFMM6HVJ5BsPQMfL/K7HhuVqBtbV5SMM0GEawDuL34LmbFKsQ/lucIgpXCSOCY1Pjg8ElxtwKbYpxpFvnarQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NYE+Aay3FUrAyLLloByMaZmI4Qgq2DLngBI4ghqmorM=; b=DI61n69/aCIqCw+K9H8QaObnZItW5wSTWWlIquV1czxXgAS0ADswyLBjt1wZQS2OrqWwDC3dQNpySkT0Y9o5MiurpdeLBuhYgR7EFhn/jBi9OWCCThS1lkh14/ytmcTCkLhiT8k2cODPkuUMWPApSDrkiItW+t8VBU2Ryx3WYoby7gIUGDNVOpU43OyyC82AzVqatvGtSV+olX3BuEnFjGX5pWV1nJw8TAKMqjooJ2nuRr0VbETneIJz+ZTX3SvtQL/CEuI4RfBXOreGxV02WdDm969GE7wvIdccEUfpEWZuEPqmt4qknYbfv9+ibXYZXVbR+EzUCFmwN/qbQqIw+A== Received: from BN8PR04CA0055.namprd04.prod.outlook.com (2603:10b6:408:d4::29) by DM6PR12MB4219.namprd12.prod.outlook.com (2603:10b6:5:217::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:44 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::f0) by BN8PR04CA0055.outlook.office365.com (2603:10b6:408:d4::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:29 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 15/29] nvkm/vgpu: introduce FB memory allocation for vGPU Date: Sun, 22 Sep 2024 05:49:37 -0700 Message-ID: <20240922124951.1946072-16-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|DM6PR12MB4219:EE_ X-MS-Office365-Filtering-Correlation-Id: ff7c44cf-83fd-42c8-e533-08dcdb052c3a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: WNrmRG1cBUBub4DPI1yKf8osMEQrxqxEmLezW0L/X1JThSyI2gsRMvtfiCsxrIBYeMdd655BY8h63BMONU0tBxx3quoURE3754YbNWnTDnDXeFLD+AuRYbulXQ4djTzov2XOaUq6djOXNfaCke2Rxtaq4T+Nwlm6xteV1WjtLioADR4Tk1fjvIL28v2y91G1pKJzJpXyG5GFCsGxF1bQ0aayi396NpNk0oJ37ukbAtm7H0uQ5ftSAMW/J3LlJIsmrt1Ngg6eGXrS63d6U1ZpVCOndPrlVhZUDdZ9kW5c2fly++BxawFCVJbOFiqRc9XHdJvKPWwg2OP7pvy9DVVgy6qlxhidUe7JTAyvMcrUVMO8srwUfcAp/+xJV5ODM8Xb8rXHkBAP8k38RvI3h7ToV5+nL+uASjgzKugy1jMR5ucMiTkBn9lUH7SnE9K08BazoIEZsoMAfnOD8RbRPUy46wedrIA0aSJozrGQjLP1XGVGuaDdDzK3vQOVTDxOuuIdZ6jaXJOSF2D1DcSlRy1d26aCsCx/y4UY5OPkV11hNYq5el9lDiKxtER1fvkBzXFJ626/a2FXx5ufl5nxVHAcTqWHDU0MHZT5nTImaLVsbQBfEh98Vj59saORqNXvobq4oT8xXTLpPhUvwJsGYTUaW24AWfEC3zFLx3tkGwCPB0t8M+mEXaiHq5kqKsN37fZT0nEGesbOW9RBl8TyST16TVqZLAl76gqE+BpTTXyfLyeSo40Ejpp6reGcvrwGuAaWoYnC66V/sojN0zlwoPZ3mL02Y2rWyj9yrvADCo1wZTheA12wfSk98cRW6zcCDtNOsi1tjH7g67EHcfVg8N8k81VldBdw/ATydTzuyVi97M2euxks9t3PFPyrIhR8hmh317J9viWNZY9uCptPlZHI/HiS+7o/aBdOpEg9msJrbBRMlgCaktTpNL6W6djQjd69jqwAi9JpdsEJnGH4sNuD1TluQ5VDgqHP9KOkpDatcSzqX/vpq/ZvLHmTcz2BjJ0TIJ4pJSB+aduLCgODRJ4uVas4fKCkgz8tlKT5RfiB9vJ9zFsFh38LQxQ9PTUyy8Dr9PT8WYXYcnsmJEmoAgDWtuhLNkiMe7wf70DmJGiHNe3XTWGlr7yMojK/uWzxgga/gXKSTnsvQTStsuoFeXQTA56J+cwrtTto8kwDqCi3ZgpPmDjg0IbNQk8d8B2wIfL3oBuKTZsj9lwEQiyKBgOeLLghyDRH/zaJSeytHaYuJDWSpNUdqSqQy5oS1leI8WjPRBbEHu+sMAZMZzkVma3feiNCJ//HXRIfLZrf5u654CvmZ7FoidGuW+B/RxcPw3ynjvFXH7/dCu5rLyw24Y/ABjWW2cVOm/ruDHhfrqHI128aBBPEp2nqeETs0+gSb/XJvXecvTIe5TrWYkgbocruELZUeZCnHrYyAdiYElzmAXi4mHUwoR8Niji8GCLzZsdb X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:44.1113 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff7c44cf-83fd-42c8-e533-08dcdb052c3a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4219 Creating a vGPU requires allocating a mgmt heap from the FB memory. The size of the mgmt heap that a vGPU requires is from the vGPU type. Expose the FB memory allocation to NVIDIA vGPU VFIO module to allocate the mgmt heap when creating a vGPU. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 6 +++ drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 51 +++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 8 +++ 3 files changed, 65 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index a351e8bfc772..b6e0321a53ad 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -6,6 +6,12 @@ #define NVIDIA_MAX_VGPUS 2 +struct nvkm_vgpu_mem { + struct nvidia_vgpu_mem base; + struct nvkm_memory *mem; + struct nvkm_vgpu_mgr *vgpu_mgr; +}; + struct nvkm_vgpu_mgr { bool enabled; struct nvkm_device *nvkm_dev; diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index 44d901a0474d..2aabb2c5f142 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -154,6 +155,54 @@ static int alloc_chids(void *handle, int count) return ret; } +static void free_fbmem(struct nvidia_vgpu_mem *base) +{ + struct nvkm_vgpu_mem *mem = + container_of(base, struct nvkm_vgpu_mem, base); + struct nvkm_vgpu_mgr *vgpu_mgr = mem->vgpu_mgr; + struct nvkm_device *device = vgpu_mgr->nvkm_dev; + + nvdev_debug(device, "free fb mem: addr %llx size %llx\n", + base->addr, base->size); + + nvkm_memory_unref(&mem->mem); + kfree(mem); +} + +static struct nvidia_vgpu_mem *alloc_fbmem(void *handle, u64 size, + bool vmmu_aligned) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + struct nvidia_vgpu_mem *base; + struct nvkm_vgpu_mem *mem; + u32 shift = vmmu_aligned ? ilog2(vgpu_mgr->vmmu_segment_size) : + NVKM_RAM_MM_SHIFT; + int ret; + + mem = kzalloc(sizeof(*mem), GFP_KERNEL); + if (!mem) + return ERR_PTR(-ENOMEM); + + base = &mem->base; + + ret = nvkm_ram_get(device, NVKM_RAM_MM_NORMAL, 0x1, shift, size, + true, true, &mem->mem); + if (ret) { + kfree(mem); + return ERR_PTR(ret); + } + + mem->vgpu_mgr = vgpu_mgr; + base->addr = mem->mem->func->addr(mem->mem); + base->size = mem->mem->func->size(mem->mem); + + nvdev_debug(device, "alloc fb mem: addr %llx size %llx\n", + base->addr, base->size); + + return base; +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -168,6 +217,8 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .rm_ctrl_done = rm_ctrl_done, .alloc_chids = alloc_chids, .free_chids = free_chids, + .alloc_fbmem = alloc_fbmem, + .free_fbmem = free_fbmem, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 001306fb0b5b..4841e9cf0d40 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -16,6 +16,11 @@ struct nvidia_vgpu_gsp_client { void *gsp_device; }; +struct nvidia_vgpu_mem { + u64 addr; + u64 size; +}; + struct nvkm_vgpu_mgr_vfio_ops { bool (*vgpu_mgr_is_enabled)(void *handle); void (*get_handle)(void *handle, @@ -37,6 +42,9 @@ struct nvkm_vgpu_mgr_vfio_ops { void *ctrl); int (*alloc_chids)(void *handle, int count); void (*free_chids)(void *handle, int offset, int count); + struct nvidia_vgpu_mem *(*alloc_fbmem)(void *handle, u64 size, + bool vmmu_aligned); + void (*free_fbmem)(struct nvidia_vgpu_mem *mem); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809104 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2061.outbound.protection.outlook.com [40.107.92.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F34E12B9B4 for ; Sun, 22 Sep 2024 12:50:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.61 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009451; cv=fail; b=G3pjNm4YjA1365sEmuKeH+TxHyUy13crju+TgpeMJtwKmNTQiWsE61T4Gid7pLK5vFDcQvYBdxnQmzwD7OoKZLKgH2KI3J9AncVzKGzw00X2KC2ooCP1CmvCeIY4PMI2yhZHeJbQDebyFhlBZQMaPKBfNd+KAnwKpyeyqDO1bSM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009451; c=relaxed/simple; bh=Ps5eoVdtyCaKVvzSntQcnhA1Drhy/P2k6iIMfHY9LBQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=APbIcYLb9NMyQw1IYhb5aSPARcMLFja65FJEHGDhBZKR7sK2X9RepF44SoJNhLg5rQOXQGUA0Hl2E1FINTpjMVcqeW81tN/q36BeGQNpZky3QCk3ApgZbYmXwuGN9WO3vtlZhgT6sDURNCDdtDXKiN3JogvxeUmhOVjogbjV+FA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=giKiWKdO; arc=fail smtp.client-ip=40.107.92.61 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="giKiWKdO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CA3PC9roIOUGYoxpMBF7E46W30OUWnaapG3+b4tcqHk7vmT1FHEm5rbRmxbQu+f4R9MKkOMGzcVurD9nz6M7Wbm0FGH1aYNrE/JdzhcT8g86iNMePHNoHu1paelY56aayQ93m9MLXTVNXLS09Y1EquCl+Zyp60NB3g2b2lw5cRDTWtkN3bA7nGjI8/3tt1/QwNSnLnw/S2KgPSVNMGWImzZMyYbNQh+sXOt2JrRnfkptXWD6hk5gI4qeT+TUp9gYYnx4Dj4yBqqDB/BMF/T4/0SuByV3uS1b+r98q3Iz4/RM4jZHegk+T7w8kpbyXw4rIppeJHwxy1YMBI4qciqXWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Rh4IiVDFgJAG1o4yb1ZHyq6rwEl0kAOJyiNfdUMFfX4=; b=cgLBwrzTPjT/TpYTeVKNJ8kCvJqcC/XlHPAN4LK+m7qY5Zw6NFpQG3qq9uSOfzrTlM1oTGezh7enRFF/XSTkrPf/tpd+M+ajEf6PT+C3H1fCzlapHNo9fUW3buIzyimpPnKumdNPUKiTQ6OsTgWNZKed3DGu20JAG02OmK3KcC/KhsjtFpWjPFjW2C6D7MLplxbNNoBZUSNRaVRKT8kslOUcRXjkTKBmmUhMcCi2SQ4aWaW3j4DICKLYb/dZ6BaRKh3t47Xn1UgbuV0Dhlqkw7+CDRVI4MTaHd3KLiFGgq9UTlxsEScb4Y7vYqpn0Lu7uJrAbdSjL6sOWiuaqVs/tQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Rh4IiVDFgJAG1o4yb1ZHyq6rwEl0kAOJyiNfdUMFfX4=; b=giKiWKdOAgkZKYsBa+L5RlVnQn4LDlw/aP94OqnK2Zf/93JLQFbU7qitGty+GiC0W5c/RIIpNQZ6zynIg/yCFsboLdDWIU9YQVhiic5sw0nEIsGmPYoR/jUXUErqK0K/Z9ntO6nb0S0KXKzCb/oTko64LYtvx/eej+B92Sh+S20bg6ZNXmox3uHvWv6JVfkJKDx5UCIoZ4VgNc4L/CGvRIlXeWyNT/Ukbg36Qjreo1S5ZNqJsmz569ByR8bGdfy3hjxBSMFY99eYlfO29SefjsH6TpciGdH8t4pWZpJxkhEKA9b5J0MavULSjIf9yk1p0i1KbY4QG9HZmpdN8LgC5Q== Received: from BN8PR04CA0057.namprd04.prod.outlook.com (2603:10b6:408:d4::31) by SJ2PR12MB7798.namprd12.prod.outlook.com (2603:10b6:a03:4c0::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:46 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::c1) by BN8PR04CA0057.outlook.office365.com (2603:10b6:408:d4::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:45 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:30 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:29 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 16/29] nvkm/vgpu: introduce BAR1 map routines for vGPUs Date: Sun, 22 Sep 2024 05:49:38 -0700 Message-ID: <20240922124951.1946072-17-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|SJ2PR12MB7798:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d676868-f146-4d67-63b9-08dcdb052d1d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014; X-Microsoft-Antispam-Message-Info: lOKslr13TJQV5Cspl/jRJkjCv0bFy72G5Ir9VEtHjQxB2gLC1/87urjQ766HBZd0uKkE1BI10LyHzdo56boJqYaVJ5zts/rLqBdVX1RYEgirE/d8dNa109w49NVCn7JhngOYxdBY14i3LQe9Zupwpl0nHdq5zTP7UR6xkX3ak7XuvEnAAcSY6Rz0dq94no7QmPXNkpFwNUjPQOox/Vh3CPYtluX+9fsqGYmsDHO6sTSJrxzZH1wheJfb1nbgehPRbXfOFERKdELsxuNNoXz7tPCfEMSdn/daCkyQDxK0ZpS5Hbd1UwtIVAtIeY13aMmsudtg2qib15sp0Kxx8QT/b72DCLfFtQDgBeh+XOh7Jf3j3l0O8ihDRWhgjE8FtmxOYUAx+Mmtem12OW0jDpaTtus0OFll6NlZm/V8u1WZmJb8Boe0pCuFw36GPEe/sAaW/aHslAt/hYD0+cl6MsxDACU21/nEk3ypJkIK9GmP2DnJPaIH+b/I9ERkiWW6pWOfRhJyYgUz27TvYjitBbbIWxAkvJkQTvEwUXw5EaJ6XEMYxCfUOJTbrmSizsJB/BeTkcE2+kynSA272T+S24deAZ2e8hHmytSu/u0Q0Mf3/we7Cyl7lc0d2pH/OFMMOasdJ+NghN6YCCtSPBpHiDrSfsgiaYt3BzTpuWKlQrq44wdWr7nwberznNKdgCnVAGik26dpxhCnS/DnklMKO9el5B2CWyLrKuFTWgtZ8DS3iU7hWcnSOR2GHuMKJhRfF3lwIawJwfoNulet9QeCnE3+1iSmKszv1wnHT9R33dzE/7vC6/QgVo0za/TaVtAqyAFAeE+7xF2X18dbcQ/QQ47/c6Z1nDanf4YBblcZmBJlBOm/GqYD4zz3uMrqjfCCCLCFNJl1mguG5xObYleDHpAeq5SvjIKASFqlkEYgKg4LM3Rv9DBEs472wfpAMv5txsjxA0m428WT29ZbAYr5ZfVvevDDNFa+o5RMQqp4U0ajja7U9sg3Jrl1uBXTJx66UMFoseQdubkHyb1pqgVhP96cYB4MaOl1qV7ntulSF+8Ra4caUNAQDkPmJ17M+klZNl8GROfdpfH8dJrZ78yXKngZCFvObjpGTKvcUnhrdHsXhdkFBdhv746O9IE1VWQcAadcB5lzRKx4/vpfnVl2J0wh5rfrUMK5cF9zuSSbQh9XU/65KNgYnm/et+ORcCJMCr7a13GQk4etiGbBBw69AWvb8Pf3ffW7m885dxVa0e/b9iOi1M5CtEUo3bQhhX1oJpld/PiOEsYi8EPZbDiYtVIVH1G2pR3gFiOxxG9FxIxTjw3m6kaAohjJ4/G5YQTbVS6ktmmopL1MKCfL+jBZt6qvmllvCVyM49fRxcj2NojPFs2SQ5a1nV5pRUndgBRvG9iSr+9SKjMgBuFiBW2psgsd7IjqEkxbpmHeXdDMGV4+vtjwjzrwB91G0trllWRyf4cZ X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:45.6113 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4d676868-f146-4d67-63b9-08dcdb052d1d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7798 The mgmt heap is a block of shared FBMEM between the GSP firmware and the vGPU manager. It is used for supporting vGPU RPCs, vGPU logging. To access the data structures of vGPU RPCs and vGPU logging, the mgmt heap FBMEM needs to mapped into BAR1 and the region in the BAR1 is required to be mapped into CPU vaddr. Expose the BAR1 map routines to NVIDIA vGPU VFIO module to map the mgmt heap. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 1 + drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 47 +++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 3 ++ 3 files changed, 51 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index b6e0321a53ad..882965fd25ce 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -10,6 +10,7 @@ struct nvkm_vgpu_mem { struct nvidia_vgpu_mem base; struct nvkm_memory *mem; struct nvkm_vgpu_mgr *vgpu_mgr; + struct nvkm_vma *bar1_vma; }; struct nvkm_vgpu_mgr { diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index 2aabb2c5f142..535c2922d3af 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -203,6 +204,50 @@ static struct nvidia_vgpu_mem *alloc_fbmem(void *handle, u64 size, return base; } +static void bar1_unmap_mem(struct nvidia_vgpu_mem *base) +{ + struct nvkm_vgpu_mem *mem = + container_of(base, struct nvkm_vgpu_mem, base); + struct nvkm_vgpu_mgr *vgpu_mgr = mem->vgpu_mgr; + struct nvkm_device *device = vgpu_mgr->nvkm_dev; + struct nvkm_vmm *vmm = nvkm_bar_bar1_vmm(device); + + iounmap(base->bar1_vaddr); + base->bar1_vaddr = NULL; + nvkm_vmm_put(vmm, &mem->bar1_vma); + mem->bar1_vma = NULL; +} + +static int bar1_map_mem(struct nvidia_vgpu_mem *base) +{ + struct nvkm_vgpu_mem *mem = + container_of(base, struct nvkm_vgpu_mem, base); + struct nvkm_vgpu_mgr *vgpu_mgr = mem->vgpu_mgr; + struct nvkm_device *device = vgpu_mgr->nvkm_dev; + struct nvkm_vmm *vmm = nvkm_bar_bar1_vmm(device); + unsigned long paddr; + int ret; + + if (WARN_ON(base->bar1_vaddr || mem->bar1_vma)) + return -EEXIST; + + ret = nvkm_vmm_get(vmm, 12, base->size, &mem->bar1_vma); + if (ret) + return ret; + + ret = nvkm_memory_map(mem->mem, 0, vmm, mem->bar1_vma, NULL, 0); + if (ret) { + nvkm_vmm_put(vmm, &mem->bar1_vma); + return ret; + } + + paddr = device->func->resource_addr(device, 1) + + mem->bar1_vma->addr; + + base->bar1_vaddr = ioremap(paddr, base->size); + return 0; +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -219,6 +264,8 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .free_chids = free_chids, .alloc_fbmem = alloc_fbmem, .free_fbmem = free_fbmem, + .bar1_map_mem = bar1_map_mem, + .bar1_unmap_mem = bar1_unmap_mem, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 4841e9cf0d40..38b4cf5786fa 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -19,6 +19,7 @@ struct nvidia_vgpu_gsp_client { struct nvidia_vgpu_mem { u64 addr; u64 size; + void * __iomem bar1_vaddr; }; struct nvkm_vgpu_mgr_vfio_ops { @@ -45,6 +46,8 @@ struct nvkm_vgpu_mgr_vfio_ops { struct nvidia_vgpu_mem *(*alloc_fbmem)(void *handle, u64 size, bool vmmu_aligned); void (*free_fbmem)(struct nvidia_vgpu_mem *mem); + int (*bar1_map_mem)(struct nvidia_vgpu_mem *mem); + void (*bar1_unmap_mem)(struct nvidia_vgpu_mem *mem); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809106 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2055.outbound.protection.outlook.com [40.107.223.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3B032D052 for ; Sun, 22 Sep 2024 12:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.55 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; cv=fail; b=KWnPap/qvdVloM/ioEGCuwR1FXph+SoU/zcmaiQYwSxyDRfGTv5ZjUMwbamYeEUv0AlhazryoIH4SE/NsbAWew4sfTwRqSTzjRU1+3FiWgzjAPhIJ1+2cxP6pV0hRAnnVN4C2M87j/b2M3a3AkC+8pZDgAGkcmj3JFiec9CbT4o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; c=relaxed/simple; bh=16nA++vFQUZKnxcPiXgHkPJ3k1gqooZg263zvgvKA1k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Akwt3UStyw3GGITTfTXEtOM5iWOKIW9t4AzU1yUoVh8pI0fMeZJFY+DZMZ7bftJ1memWyMzE3TNEjIa9/CtUDZGGUWhbih3s3b9HX80bFutCv9nUtwWuwfLrieuhm45X0tdzFZVZZiWxxdjh8EDWiW+bCY1eKBS4LROcRbiVFIw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Niii5kAE; arc=fail smtp.client-ip=40.107.223.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Niii5kAE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gdcFWWbJjmHkTkWToojzyf9cc8+YoA80XGls1cVbTKcAF/AE33JBTWoUsB+3Cz3rZjOecQAYId01DSRLorgHUrIoiCC6+2P8mZGnHoIw1NoM/gArSv1d7GrYpsZHeVueokFFpYZqpvvUFC3ZUlgN2ojSS8dUaFVsGrvkXvdL0ww5yMz/Out5pLgFnUf/oQNcpAhmCifmxOO2Bh3ikBQtDcBkLwSDfBbkW03pe/djLruZdODmAMtm4vpHcYEWvrp1pE/69hfR6Ib+w9ZoZ0acl/94rYG/LCze0Bqt+7vZTrzAvn0oznLeCV6lJo/xlPvSyxwNoDrGxT5rdMgVfobhrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CMa2Jz9AWsMojiEGyYYPttR9zgW+Jcd1tgQ2HKC/MoA=; b=V8XHy+5AhX+XIlnql9vOrl3rB/XpP/uFafRS5LsDK6MKlGMrR/PpGNdhBGhKTHxvtitLSjO8ddn9Cd0R0M7OHl0NxgxDv1RgMc2whH0I0Smq/FZS/rh8Bls8aDUoWL8ecnDdLlb5+L5y6lqGS+TRDpty9us6FRiVNUFEyIhn81gYdal8zjVPK8j6/tIR1LEapiFaymg6y5bBz4KEjVm+NRwh+VNty1APNWvdleYT9PdyPbaUOemE6Pex58NIWrJ7KusyqRno3g51JQkk/FtCF6+hVPXJQstHd4dKdcVMcnFJtVIAES4Yfv0M9lAwaHTSVnYLtTpAX1adgFU8sRxIkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CMa2Jz9AWsMojiEGyYYPttR9zgW+Jcd1tgQ2HKC/MoA=; b=Niii5kAEkZU0V+0GeI1VEu9Gonc0NZjgO2uyrm+D2ZoboM3B6s2/kyQkbxzN3i9QMDlWtWrqb5O3nnoSKkoxgpJdZz2FYcJt6mM/iD1xKM+ZOkXMu3CFjhGKKzKgnQux36gd1W4J/ggp9Wp+2dtGYsMs3rMmAVya+2KLihRBe03PObvlDa2L+LYb0VTgzO2cMhxlAtP264/Ughm8S4naUZg4TqWwVX9QVIRv4ETy1BXa1EZAk3Yr6SQfIJ0zONyv/S4JmiyPpq7kiM8kLHbrFwtkFluQ4KxC4zi+MLH6/4UKJCzKlX/HEAfdLrewOjK0ZcXY+nPvXuWjyx0bDMmyZw== Received: from BN8PR04CA0039.namprd04.prod.outlook.com (2603:10b6:408:d4::13) by SN7PR12MB7105.namprd12.prod.outlook.com (2603:10b6:806:2a0::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:47 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::2f) by BN8PR04CA0039.outlook.office365.com (2603:10b6:408:d4::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:47 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:31 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:30 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:30 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 17/29] nvkm/vgpu: introduce engine bitmap for vGPU Date: Sun, 22 Sep 2024 05:49:39 -0700 Message-ID: <20240922124951.1946072-18-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|SN7PR12MB7105:EE_ X-MS-Office365-Filtering-Correlation-Id: 546ad3a8-41ed-40ac-5659-08dcdb052e34 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: 79jMkd13FD6qI3iS7R2kvB8IweXP0jXsfFu/zzR3RmPiXIlqbRzSmnZfk2cG60/jsQWgSvpdg67yUIvkNgd3LGEvBqfRqINLaMDr7arIx9FAKkM7wELpLSf0QOQTI4f3d1QCt1PGQNneRDo0HXIt4bthjuE9n6iv5szRebwMcHUZJuxYcUnYeLXQViVQikz4SBJgJzdzpYqwDIupUWnQReAk+TcnZIoOLqVJgbW4AWbhzIwMNbX03Xktz98bNmQPXb55gh9EBPKHD3EYeyhkolL1xLBHbuLWX4uwMk2u0ki23w64c/tP1g0sxYtqRfuCwcrOAsytMCEUOEtgsVgqOo539oY39PnE6glNbhHMZ4qVaQwaVZ2jDi11rsdKxkJXc1AEfwkTTPoKU36KWSIUjm0TwlxSvZ6tfYry80tD8lvcNJjIMSgOpovlhrgqJfILzrXlhFKblv7E5AI5Gu5dwKUvTTWbrAYzIHgXRZuu79GFOyxmIU1LTPN0efn6Z49+3t8nROxg/GI/86Z/RuXvWhtDF+mrjLCIXyb/S3Dxj9eE91rgW7EwoOTH4p/nYGwe27iN8mkHjPHTe9NdW12qZcScn8NNxPhiRZ94ESgo5KqjKXXR2x8pFLn8md04/66vwR0mdy6xbFFZ24sx2RRA8fbkDjBhTkWTGeg17NJJLQIVH8eQHu7iHWEd6R5cMwqB84IMrt03q89RxIwkFOf0yBfCfa53wbRYFU00cqFXE16KOyat8ytN1HOpwM7XC/0Ti2YSm51D60VzRYxZcjSJZxdm2aJuGgPPiui/5aRfXslRYvi89Rhj7C3DzMusqTDns7Qkywh9kGwssuKcTPzEyxFjAF8H2Ow4+oseazrYVTGebjpOVcRFIjzv69X/3DMN1dU8GgosHbxPznxc7tLXS3QkUaq1cUvHxGOKuieZ0R2b/iRJB6HeY7sNnVjQgqjAbp+M4wr0p079XTAsjxxFpGvHEKyXnr9bPkUcYNpc5iCgbBDU2B2mZyGP7QNfGUt6fH93twjFVZXEou+gK1fISgzgSU8Ozzob8ZNYet00t3OIgZ6Nzr3JLJNaSqj4sAhv98Z2pfY9m4+g/9k+fjvgCxNBvPX8vJ4Ogd/do+q9TyF1EC2gbabpnC7PRmEZWI3Knj0qPQEAsamkSTP80bVZ2Dyb72DSAkPVgokCqzwd8Ji8ANnPDiWugS0x3ZMPRxuY2nq6L2HNBC54vB4NH+PkSKyXpkMeTcZ6iPhihakxaSiWp59TWQcEQ/3LiMG6DeWgCZ640h6aJGhQKO/jtPpYLkdCUj0mFOEzCFhanOmlEmztQNDgbHOVNefQdP8WQ6WEig/W1alIi8VdzYUlG3/sJ5y7E8wff3uFbzpMvs8gJcBeY8MbT0p+A6N/bhAYo7MavtYB5YXbJG+NLQTgHJatkTJDFlbbUMC1Cd7l7HnBhvZLNMM8E0PDEQYPJEFQjOZn X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:47.4238 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 546ad3a8-41ed-40ac-5659-08dcdb052e34 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7105 Activating a new vGPU needs to configure the allocated CHIDs of engines. Thus, an engine bitmap is required. Expose the engine bitmap to NVIDIA vGPU VFIO module to activate the new vGPU. Signed-off-by: Zhi Wang --- drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 15 +++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 1 + 2 files changed, 16 insertions(+) diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index 535c2922d3af..84c13d678ffa 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -248,6 +249,19 @@ static int bar1_map_mem(struct nvidia_vgpu_mem *base) return 0; } +static void get_engine_bitmap(void *handle, unsigned long *bitmap) +{ + struct nvkm_device *nvkm_dev = handle; + struct nvkm_runl *runl; + struct nvkm_engn *engn; + + nvkm_runl_foreach(runl, nvkm_dev->fifo) { + nvkm_runl_foreach_engn(engn, runl) { + __set_bit(engn->id, bitmap); + } + } +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -266,6 +280,7 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .free_fbmem = free_fbmem, .bar1_map_mem = bar1_map_mem, .bar1_unmap_mem = bar1_unmap_mem, + .get_engine_bitmap = get_engine_bitmap, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 38b4cf5786fa..d9ed2cd202ff 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -48,6 +48,7 @@ struct nvkm_vgpu_mgr_vfio_ops { void (*free_fbmem)(struct nvidia_vgpu_mem *mem); int (*bar1_map_mem)(struct nvidia_vgpu_mem *mem); void (*bar1_unmap_mem)(struct nvidia_vgpu_mem *mem); + void (*get_engine_bitmap)(void *handle, unsigned long *bitmap); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle); From patchwork Sun Sep 22 12:49:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809112 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2048.outbound.protection.outlook.com [40.107.244.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C94444A07 for ; Sun, 22 Sep 2024 12:50:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.48 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009459; cv=fail; b=UzLsDYmivvP9NR8Mk0LuxYtA3IBh40XCIBg7p7FhkpQxDbhxZv29fqvJ0aJc8xLoQWeuA4xoWx5b3GfJvQQJTLCX4AevKPKEavqjaXnSHZ8NWIMdnlmXP4dGgoTGA/VZPVxdTixKrulAs/D9rsI7L6nrGAZBQzenZK4kydtGTV4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009459; c=relaxed/simple; bh=77oTDYRhoAy9U4DjYei5+MOsWNcRBKHlByO/DbkoXIg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=l8rRxqo42hibFcrhDRlBmFH5B5e0liEFu7wHd0gWYZdHzCaeu5vd6ArFH2qOo9yi6caEJp+g5HOs4ZkUV0O0rSITaxe2p4VG76+mMrfSblYpudPQK+0bcEbMRJNV8Pv9qMley4KMKrRRl4aZFto4p4qVf25Q1gWRSyIsF/nciuM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Jw399xWt; arc=fail smtp.client-ip=40.107.244.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Jw399xWt" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Fvt6IdhQjtekmKUtGBGc3Hv2bSWTw0XCC2TRcPS0jAWHqjeJ+B8vjioH3PAMX0Koc/K2GL8eXXm9dIJtQ9Fiut9mylReqS8NDEkTk2Nt6vNZRY7+xwQ1rdnzXSMhRVAJowEew4fWb6gjg530Kfk4gSojHSVcWpxVtjKI8ceO7clDM5ozIAzwoA47UiV377KvpqHpaJKMyijZFaOxL8SoiAsZDaavK18ktbaOYVLSsghjevYpmmRj37eyjwJOa8fCQ0E48wb4VheSJ5M+mhnpQv1i8vlVOlghun3ga1cfum8rdehRVzbnMnSVEpJw3+8nixOX2LtHbFiNpYVEduBwjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PSPcodUi+g62DqoRwDS0XVeRMjRuWBo0CZPEqrjeZS0=; b=hcfVc/NSqIvls+JsYhLqw5Vb+XP1CGq51GIJSUjXr8uG46HZF4brIQdWXARDFVigMTX802wBvSH4K6hEQvuyO297Ni64gnHvUh3s43dVYfJ77kDXnmdeAVJVttCE1oF8XphK6Qm38cmHVcYM5pmIiAYn1q38ofFrUSYa8xC+W1ZAto0X5fp2HfnuVAma2AQJvbhZyP8YfEhGILKnR2Q37NVh6EaCKlFfkfYvQOJmB+0BE3/M7rRAU6qR0hhlTicHIaZjwElWHuy7NtZx/9KXkmwI7kD1E2Cf28ySzQP4HCoXM/STKC4hcbzvyCULS16ekXzIUq2ZAtW85k+OwuGzMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PSPcodUi+g62DqoRwDS0XVeRMjRuWBo0CZPEqrjeZS0=; b=Jw399xWtPyFTPh9cSSMoUmiRXzVcETRXeeMnnJ8Kb7Wsq0/mr59g2+guFttwxoJZBUCZFMda8V9cx1W9332HSXvo9hGAyrFExli72E/6LjaqZxkWmzGl+UzemnkXv9mpvZT/F/MpqefpdnJOCs5PCR8Ni9jgiqADu28OPNfxd9OSc6KC4zEj6j0z1dUSQqpemrxWv290iLa6J2yqIcA20nk3o004nbZZRGR2CnCKbUGacjA+OPdl/x1tk6EyDbZwYGbrjRYSz1Q/0Un5wEr59JzkcNalPGOaZpZnSu0GXz1poTPS5e/KldZlrsa+5W0ksQzhlNRyfy0tBMlKxUA+Nw== Received: from BN8PR04CA0051.namprd04.prod.outlook.com (2603:10b6:408:d4::25) by IA1PR12MB6257.namprd12.prod.outlook.com (2603:10b6:208:3e7::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:49 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::25) by BN8PR04CA0051.outlook.office365.com (2603:10b6:408:d4::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:49 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:31 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:31 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:30 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 18/29] nvkm/vgpu: introduce pci_driver.sriov_configure() in nvkm Date: Sun, 22 Sep 2024 05:49:40 -0700 Message-ID: <20240922124951.1946072-19-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|IA1PR12MB6257:EE_ X-MS-Office365-Filtering-Correlation-Id: ceaad478-48e2-4446-e525-08dcdb052f67 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: 1pOCnV5vQNf2++SlGwS9z1Vrf1VTKVdo6f5DzXx5tBXKYTT7MBA1Xu6nF0S5DAUQUO6tZP3/im7iDNmW93DYF28qHIsfhMo81yg90gZ7fFIT28Aq03Xogj29wiTJsu+dLYkEp7k1ittXda3Tfi0LCuYHPMgjE5P8Zz/nXaCQ/1fmQJQdvKSgIA5Zn6vQGfckZGqUgoVyReqTo5l7UWKs8aG7OMCB1yBOWv8FB5V0M2bJ2QU1SHn01IKhkZJpsY5e70Bil8ZB//5e2hSRVtc8+PQ43DbdbBvNgV05wIzP+vGW8jM3NTQ1t5Ci4j9xhiL4dkcT/PGFX0eBhbKWL1fo2LUZaLfuBvXJRBFRZbRMpHFwErpqV1fz+rZ6MTJSpnaj6gi6La3PRU1Pi8uqjBUGgIpaTa8no9KjmLbkwhyuOXDDu1VgYJyhHBDVyXuBdaAnilLLjiuPvpMLsGlOm5/h+0rm5vV/jNNSGw2nlutSmCl9y91zzQJ9SrtDBpiKZtw4CwQXMq6iEcfoHjmArVUzpQqGJNiOSRMQj3/vl2RkiVukMKe2fe6P89/wckZfB+2cPiezwb+ETrPk+XceSLenwN1S7TDN/DuKpATxWF3ffCubFD1Py3kPTtLKUZ9knTZZJWVL1G20gdvnTbe7mw6GHEp1bUwHvzg3ZnN4gSIgVjlElIpVUMsFS4M3zeFVVX4lQG5pZL4er8ldWWXAhHvlysjGhzKxBCdwB9Ti7X1B2/WWZtBYU9ZWC3rjCmHXQWno/U+DJflNaMLbnk8sxIw5AJO8rkezSiMSVq2DaehVx7KLEfT5acjjK2jTVw+QmAsWebNf4Up+qKOjBPRQfO0dBv46ePGippjvmauX1T6GwrvZ1b18TN/coM/TXzDvx2T1pUzBuaRVxUds/lLRFNgahIK9E1L9Y/ECTrW276m7/2aer9pQvWWeHBttf4iBKILSsRa3BcFS5jVT0dlR+egjss+0+5nHWLCs4Drp65Mdux+I4rL7+nFlggW+yXK/XMKEi6Dj/i2P09SbqOOs5ryT8YdpLjbahKyXa8/xKSYkwThMntupWu5xxRAoR6chfXIpbDqFSfg0j1nIMZvX53cnIV6M7Kf5NJy8t8RNaWAx/KK2Vk/T3bmrZkN10TGOuUxUIN55FOTWMZB3tOvBLhxgJUwPmQX1ZPxZoIWh2Bzp9BmFNFzJjy5w0vHmhwN38AS9lbfbe3JmvkHutonrG+1yOHdvAhyeR9pRAti/w8kdq5sVpyljaLaDndeKDuxE4J8Bm+IM+LN/VEkaR1AdgJis+jif18MePCKAR2TP67S8sfOpeKT3lRiCmmYjRGGCdU1+9xdUIMoHeWEsm1LYJlh4zE9TiK4bvNK0z0wu5ULPSC8= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:49.4550 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ceaad478-48e2-4446-e525-08dcdb052f67 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6257 The kernel PCI core provides a sysfs UAPI for the user to specify number of VFs to be enabled. To support the UAPI, the driver is required to implement pci_driver.sriov_configure(). The num of VFs are only able to be changed where there is no active vGPUs. Implement the pci_driver.sriov_configure() in nvkm. Introduce a blocking call chain to let NVIDIA vGPU manager handle this event. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 1 + drivers/gpu/drm/nouveau/nvkm/device/pci.c | 14 +++++++++++ .../gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c | 25 +++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 5 ++++ 4 files changed, 45 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index 882965fd25ce..388758fa7ce8 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -40,5 +40,6 @@ void nvkm_vgpu_mgr_fini(struct nvkm_device *device); void nvkm_vgpu_mgr_populate_gsp_vf_info(struct nvkm_device *device, void *info); void nvkm_vgpu_mgr_init_vfio_ops(struct nvkm_vgpu_mgr *vgpu_mgr); +int nvkm_vgpu_mgr_pci_sriov_configure(struct nvkm_device *device, int num_vfs); #endif diff --git a/drivers/gpu/drm/nouveau/nvkm/device/pci.c b/drivers/gpu/drm/nouveau/nvkm/device/pci.c index 1543902b20e9..f39d2727d653 100644 --- a/drivers/gpu/drm/nouveau/nvkm/device/pci.c +++ b/drivers/gpu/drm/nouveau/nvkm/device/pci.c @@ -1766,6 +1766,9 @@ nvkm_device_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) struct nvkm_device *device; int ret, bits; + if (pci_dev->is_virtfn) + return -EINVAL; + if (vga_switcheroo_client_probe_defer(pci_dev)) return -EPROBE_DEFER; @@ -1867,6 +1870,16 @@ nvkm_device_pci_probe(struct pci_dev *pci_dev, const struct pci_device_id *id) return ret; } +static int nvkm_device_pci_sriov_configure(struct pci_dev *dev, int num_vfs) +{ + struct nvkm_device *device = pci_get_drvdata(dev); + + if (!nvkm_vgpu_mgr_is_enabled(device)) + return -ENODEV; + + return nvkm_vgpu_mgr_pci_sriov_configure(device, num_vfs); +} + static struct pci_device_id nvkm_device_pci_id_table[] = { { @@ -1889,6 +1902,7 @@ nvkm_device_pci_driver = { .probe = nvkm_device_pci_probe, .remove = nvkm_device_pci_remove, .driver.pm = &nvkm_device_pci_pm, + .sriov_configure = nvkm_device_pci_sriov_configure, }; MODULE_DEVICE_TABLE(pci, nvkm_device_pci_id_table); diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c index 3654bd43b68a..47c459f93950 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vgpu_mgr.c @@ -207,3 +207,28 @@ void nvkm_vgpu_mgr_populate_gsp_vf_info(struct nvkm_device *device, v = nvkm_rd32(device, 0x88000 + 0xbfc); vf_info->b64bitBar2 = IS_BAR_64(v); } + +/** + * nvkm_vgpu_mgr_pci_sriov_configure - Configure SRIOV VFs + * @device: the nvkm_device pointer + * @num_vfs: Number of VFs + * + * Returns: 0 on success, negative on failure. + */ +int nvkm_vgpu_mgr_pci_sriov_configure(struct nvkm_device *device, int num_vfs) +{ + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + struct nvidia_vgpu_vfio_handle_data *vfio = &vgpu_mgr->vfio_handle_data; + struct pci_dev *pdev = nvkm_to_pdev(device); + int ret; + + if (vfio->priv) + return -EBUSY; + + if (num_vfs) + ret = pci_enable_sriov(pdev, num_vfs); + else + pci_disable_sriov(pdev); + + return ret ? ret : num_vfs; +} diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index d9ed2cd202ff..5c2c650c2df9 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -6,8 +6,13 @@ #ifndef __NVKM_VGPU_MGR_VFIO_H__ #define __NVKM_VGPU_MGR_VFIO_H__ +enum { + NVIDIA_VGPU_EVENT_PCI_SRIOV_CONFIGURE = 0, +}; + struct nvidia_vgpu_vfio_handle_data { void *priv; + struct notifier_block notifier; }; /* A combo of handles of RmClient and RmDevice */ From patchwork Sun Sep 22 12:49:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809108 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A5872AD33 for ; Sun, 22 Sep 2024 12:50:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.60 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009454; cv=fail; b=FMoXyhBMxaRklMI8a7Ad2zNR5WCJcjcSGqdDcXaV3aJUmwp9Tx3CgJ6WfnDcdVATWdEoMVIpRxKs+Z2RzwiRvNCSlmYYkeKvCTvOSfFUY7w7I8I+yw0F1Ab080i6Jn13kY5k/Jk91mtDkiKeXrbtWkdTPPDK9uu/QivIBxUFQ7w= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009454; c=relaxed/simple; bh=3zHkSPwBeY2OG33JdP2Tigc5oRE8cSo99YV/Z+gVswQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Y07bpYa/gzXf0LXAnrCrHfya7J92YZVxh4vPByQjimQyiHNv6i2/5cDWMtVgKD3XEorPrjO7aM3Qnr5eMwA++qkLrl3MBPNG3n71uyEvsdkqhNuu2YygN/6sVOnMxG2p8SZny5uf2tMGNuhr+GF1l5t8/pexUyAJVUvxN/EjXPs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=SzHBoD70; arc=fail smtp.client-ip=40.107.220.60 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="SzHBoD70" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MQ7LUhDWPzvgVLz3IYtSNUUqb0DpQ4MBxGmmapFhKVcfX8KIW7l7A/7qeXPEu0MTd1kivpR6UvuWe1MWV+ZlsbX5V/QpVMVyHwioDvrO8za1CMeAqcddoKCCtVsqOtCdu6XiPlXAtTkb2TBLKyHCm8LE3VLPnJeHQL59qI2KEpifTD0mVhWGDn+0NTgx9FeJscOyAjosve0MVr0zF+HSR+BgK09gR35v+3en5UOP4T5WCbEHWFJBrYtvIHin/SjypTRz69VXmPpfhLhH3wxHdOsgaEgeS/5M35WjQsNf+9BG7hSE+WM9YET8iWrn0/oBKaDGxzQPlenxPWFLOID4Yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tO7GDRY2U3uvpaqhcosC9SNjelzDMHBPuhdAWI2kBe8=; b=muiL1H9d94HlYJ07MbRGeEdljAwDwQAVVN0HyJ0GB8S98Y56gVm4Xq83o0458C5a7NNJt9DQaJLYPIjK64w/KmDU8ofjjFtq87qrJYCJ3zJBaxzdNz4hezItfBClpgMH2QTjwCFe8+Grr68RqHOreKlrcSpgYjyUkLHCcAEESeybZJ8MAVhX/MTZi62eVqa73TtXmi2cVV8tDUdXhlTJtaSFBwlC4zUIzeN7Swxu+CaujIFKbEVlxfY8i0YpIeWi9KxEDr8gr1keKUPPv9Vo74sUff/UOWsoD6Ok9Pns9y+yIi5tEyur3/AB8kO208nHQGPWJVC4yleYkVPfiENG+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tO7GDRY2U3uvpaqhcosC9SNjelzDMHBPuhdAWI2kBe8=; b=SzHBoD70EBFwDOsSI/wb3QowlJSXHvVey/SCc1t80o42yZOiEmVbMzRqqp9UpzEB648et8jdak0nj/WZ9SMasOW6SIyBVVhb1dvUCAN/PHIH0vLWOO6NoynrVH696ATE6ZnyHbt4btt782gc9p6DICpytqLNuSFQSGzi5StbIazyjm/PDqOSK2ME9w99lxJPizyvLBel339TTeiVKckUPwvqpHMbGoGggTOLewS5H8jJXqxEW6VyM6c9NUIlEmrNY0lsDvCvfjTeFDbOmywzJoqN7HcGrC/74CaBCVWHBGYbeRkEh1PYLp3ZcqeKvc4oWf/w78vFKJ5tkptJgAHXug== Received: from MW4PR04CA0318.namprd04.prod.outlook.com (2603:10b6:303:82::23) by SJ1PR12MB6241.namprd12.prod.outlook.com (2603:10b6:a03:458::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:44 +0000 Received: from CO1PEPF000075ED.namprd03.prod.outlook.com (2603:10b6:303:82:cafe::f9) by MW4PR04CA0318.outlook.office365.com (2603:10b6:303:82::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.24 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075ED.mail.protection.outlook.com (10.167.249.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:32 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:31 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:31 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 19/29] vfio/vgpu_mgr: introdcue vGPU lifecycle management prelude Date: Sun, 22 Sep 2024 05:49:41 -0700 Message-ID: <20240922124951.1946072-20-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075ED:EE_|SJ1PR12MB6241:EE_ X-MS-Office365-Filtering-Correlation-Id: 848d5c2d-f920-449d-8b3c-08dcdb052c24 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?q?cfmy08fyegh5RM5p7LRx559g0sUgbLW?= =?utf-8?q?ppA2c8Y3o05Tk4uIYFdtHWP04UPcKgVZyZZ/P+wkk7JTzlE2dNrEmYkEwWCZTOGg5?= =?utf-8?q?H1xIxZ6ALLD+1ZDKy0KD51L4ha2ByxPksye24HIqTXgQvD48dz32OM9nR/c1WOSqZ?= =?utf-8?q?yiUq7Ly22AVLFkKT8XZcebQJg7U1VX/OJCsuPKfIMMPaF/onri29JWXsEtFpvpogi?= =?utf-8?q?M8m+JAzyGimL791UgK5sEbiHsxOQ9b15VTZokFC/sy05uDEQlGWLwTuRMkkpt1Hsm?= =?utf-8?q?rCtrGXzb4/ADwsbwIiE54XQafFnqUM+uBWlzKCValTIDb7xCYhHy6NYucWzNH363K?= =?utf-8?q?TUUiyThi+JNKzCWlbUiVn9j3V4FvMNbs1sVBCn6+zSRFenZCrF+PXabGd5wN1ep8e?= =?utf-8?q?ZTO2yrswTpYfMdQMT6xrwFBMtoCfDsPlo8r5UI80mHWHAUmZekCxZd47e+l65ueCL?= =?utf-8?q?zSkbFUEvc3JLYzJ6O2GQx1Tw0H9NHMwblRreKPN4LECmtdpUtqx7Lrvy3R6NyQp2K?= =?utf-8?q?3R4dCmLn96h/jjCiFAiY4radny70nxE7ApSlL/6saNgw9jP3n6Wb1jRbUe73Vgb6q?= =?utf-8?q?NyWN0ldoEwAJx2FqNdevXe8Oy8s46Is+58DkDKJkth2OL884eAHRLorp/VschOLiz?= =?utf-8?q?4S1QbcbzVBGG1jep5DY9lEyFFTlW4siTcTLrTyV2UZsmOf8wfqrWLTipkOf3ffELy?= =?utf-8?q?j9J7eivqPEq4/K0NJm9ZJgQEmza4gTUugLQP0W3W/AOHr22vnvtfgEKQUb8v6CLsY?= =?utf-8?q?+e2Amk1bN2KaXvXeC9hphbf33Vmb+iTc1LShAvsOgS7Yc0ZN9Qx2gVAAqcsZnGcRa?= =?utf-8?q?uHhNF9aOeJv47JoQeO+k+kpciQEBWR21r3K43WyQ2nUkd23BZA73g1dWgjyy4ABSX?= =?utf-8?q?rVtOSRhHJYzdYGX18MoS/V/bm5eZVP1gtPcX8BH5sMjfbtpn5Efwkw1Y7S84PVFFt?= =?utf-8?q?8U4LHrvkT2m3tPHCljNNiv4lwsT6ZKdG/ormiToDzKRTmWLV+y33t703rLbMlSXWV?= =?utf-8?q?IUR0IRiLeImKcNwkIsJqO6vo9tsXtJlzLa+uZQjijllhkjAkmdoPCRiRns8i1Geee?= =?utf-8?q?nWn242PIVzDLIBpP29s074SLLL5qukvVrpoWYmAc2q8+d1efZNhQpEFza1LfbDmLI?= =?utf-8?q?P48+Ly+QGo1m+vR/ik3y9aObnSfQjxyIrFjJ0JZTZlprGG1x5nG/voI5k1K8bOr6B?= =?utf-8?q?+76M/bGmwRQly4hpYcOiMjQBwEtecLVBKtey69gsYJH0iFO14lNTSGV3cDvIWFN1r?= =?utf-8?q?dXu5DH8Xw9yfUOusz0dIENaMNjJ93vdXijB2WhaHXB6vVc4aDSR+S3rS8/Z24Ibpn?= =?utf-8?q?6UzCwFQbTUnr?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:44.0897 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 848d5c2d-f920-449d-8b3c-08dcdb052c24 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075ED.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6241 To introduce the routines when creating a vGPU one by one in the following patches, first, introduce the prelude of the vGPU lifecycle management as the skeleton. Introduce NVIDIA vGPU manager core module that hosting the vGPU lifecycle managemement data structures and routines. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/nvidia-vgpu/Kconfig | 13 ++++ drivers/vfio/pci/nvidia-vgpu/Makefile | 3 + drivers/vfio/pci/nvidia-vgpu/nvkm.h | 46 ++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 83 +++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c | 99 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 43 +++++++++++ 8 files changed, 291 insertions(+) create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile create mode 100644 drivers/vfio/pci/nvidia-vgpu/nvkm.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/vgpu.c create mode 100644 drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c create mode 100644 drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig index 15821a2d77d2..4b42378afc1a 100644 --- a/drivers/vfio/pci/Kconfig +++ b/drivers/vfio/pci/Kconfig @@ -69,4 +69,6 @@ source "drivers/vfio/pci/virtio/Kconfig" source "drivers/vfio/pci/nvgrace-gpu/Kconfig" +source "drivers/vfio/pci/nvidia-vgpu/Kconfig" + endmenu diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index ce7a61f1d912..88f722c5c161 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -17,3 +17,5 @@ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/ obj-$(CONFIG_NVGRACE_GPU_VFIO_PCI) += nvgrace-gpu/ + +obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu/ diff --git a/drivers/vfio/pci/nvidia-vgpu/Kconfig b/drivers/vfio/pci/nvidia-vgpu/Kconfig new file mode 100644 index 000000000000..a9b28e944902 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/Kconfig @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: GPL-2.0-only +config NVIDIA_VGPU_MGR + tristate + +config NVIDIA_VGPU_VFIO_PCI + tristate "VFIO support for the NVIDIA vGPU" + select NVIDIA_VGPU_MGR + select VFIO_PCI_CORE + help + VFIO support for the NVIDIA vGPU is required to assign the vGPU + to userspace using KVM/qemu/etc. + + If you don't know what to do here, say N. diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile new file mode 100644 index 000000000000..1d2c0eb1fa5c --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_NVIDIA_VGPU_MGR) += nvidia-vgpu-mgr.o +nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h new file mode 100644 index 000000000000..4c75431ee1f6 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ +#ifndef __NVIDIA_VGPU_MGR_NVKM_H__ +#define __NVIDIA_VGPU_MGR_NVKM_H__ + +#include +#include + +struct nvidia_vgpu_mgr_handle { + void *pf_drvdata; + struct nvkm_vgpu_mgr_vfio_ops *ops; + struct nvidia_vgpu_vfio_handle_data data; +}; + +static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, + struct nvidia_vgpu_mgr_handle *h) +{ + struct pci_dev *pf_dev; + + if (!pdev->is_virtfn) + return -EINVAL; + + pf_dev = pdev->physfn; + + if (strcmp(pf_dev->driver->name, "nvkm")) + return -EINVAL; + + h->pf_drvdata = pci_get_drvdata(pf_dev); + h->ops = nvkm_vgpu_mgr_get_vfio_ops(h->pf_drvdata); + h->ops->get_handle(h->pf_drvdata, &h->data); + + return 0; +} + +#define nvidia_vgpu_mgr_support_is_enabled(h) \ + (h).ops->vgpu_mgr_is_enabled((h).pf_drvdata) + +#define nvidia_vgpu_mgr_attach_handle(h) \ + (h)->ops->attach_handle((h)->pf_drvdata, &(h)->data) + +#define nvidia_vgpu_mgr_detach_handle(h) \ + (h)->ops->detach_handle((h)->pf_drvdata) + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c new file mode 100644 index 000000000000..34f6adb9dfe4 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -0,0 +1,83 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include "vgpu_mgr.h" + +static void unregister_vgpu(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + + mutex_lock(&vgpu_mgr->vgpu_id_lock); + + vgpu_mgr->vgpus[vgpu->info.id] = NULL; + atomic_dec(&vgpu_mgr->num_vgpus); + + mutex_unlock(&vgpu_mgr->vgpu_id_lock); +} + +static int register_vgpu(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + + mutex_lock(&vgpu_mgr->vgpu_id_lock); + + if (vgpu_mgr->vgpus[vgpu->info.id]) { + mutex_unlock(&vgpu_mgr->vgpu_id_lock); + return -EBUSY; + } + vgpu_mgr->vgpus[vgpu->info.id] = vgpu; + atomic_inc(&vgpu_mgr->num_vgpus); + + mutex_unlock(&vgpu_mgr->vgpu_id_lock); + return 0; +} + +/** + * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance + * @vgpu: the vGPU instance going to be destroyed. + * + * Returns: 0 on success, others on failure. + */ +int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) +{ + if (!atomic_cmpxchg(&vgpu->status, 1, 0)) + return -ENODEV; + + unregister_vgpu(vgpu); + return 0; +} +EXPORT_SYMBOL(nvidia_vgpu_mgr_destroy_vgpu); + +/** + * nvidia_vgpu_mgr_create_vgpu - create a vGPU instance + * @vgpu: the vGPU instance going to be created. + * @vgpu_type: the vGPU type of the vGPU instance. + * + * The caller must initialize vgpu->vgpu_mgr, gpu->info, vgpu->pdev. + * + * Returns: 0 on success, others on failure. + */ +int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) +{ + int ret; + + if (WARN_ON(vgpu->info.id >= NVIDIA_MAX_VGPUS)) + return -EINVAL; + + if (WARN_ON(!vgpu->vgpu_mgr || !vgpu->info.gfid || !vgpu->info.dbdf)) + return -EINVAL; + + mutex_init(&vgpu->lock); + vgpu->vgpu_type = vgpu_type; + + ret = register_vgpu(vgpu); + if (ret) + return ret; + + atomic_set(&vgpu->status, 1); + + return 0; +} +EXPORT_SYMBOL(nvidia_vgpu_mgr_create_vgpu); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c new file mode 100644 index 000000000000..dc2a73f95650 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c @@ -0,0 +1,99 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include "vgpu_mgr.h" + +DEFINE_MUTEX(vgpu_mgr_attach_lock); + +static void vgpu_mgr_release(struct kref *kref) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = + container_of(kref, struct nvidia_vgpu_mgr, refcount); + + nvidia_vgpu_mgr_detach_handle(&vgpu_mgr->handle); + kvfree(vgpu_mgr); +} + +/** + * nvidia_vgpu_mgr_put - put the vGPU manager + * @vgpu: the vGPU manager to put. + * + */ +void nvidia_vgpu_mgr_put(struct nvidia_vgpu_mgr *vgpu_mgr) +{ + if (!nvidia_vgpu_mgr_support_is_enabled(vgpu_mgr->handle)) + return; + + mutex_lock(&vgpu_mgr_attach_lock); + kref_put(&vgpu_mgr->refcount, vgpu_mgr_release); + mutex_unlock(&vgpu_mgr_attach_lock); +} +EXPORT_SYMBOL(nvidia_vgpu_mgr_put); + +/** + * nvidia_vgpu_mgr_get - get the vGPU manager + * @dev: the VF pci_dev. + * + * Returns: pointer to vgpu_mgr on success, IS_ERR() on failure. + */ +struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev) +{ + struct nvidia_vgpu_mgr *vgpu_mgr; + struct nvidia_vgpu_mgr_handle handle; + int ret; + + mutex_lock(&vgpu_mgr_attach_lock); + + memset(&handle, 0, sizeof(handle)); + + ret = nvidia_vgpu_mgr_get_handle(dev, &handle); + if (ret) { + mutex_unlock(&vgpu_mgr_attach_lock); + return ERR_PTR(ret); + } + + if (!nvidia_vgpu_mgr_support_is_enabled(handle)) { + mutex_unlock(&vgpu_mgr_attach_lock); + return ERR_PTR(-ENODEV); + } + + if (handle.data.priv) { + vgpu_mgr = handle.data.priv; + kref_get(&vgpu_mgr->refcount); + mutex_unlock(&vgpu_mgr_attach_lock); + return vgpu_mgr; + } + + vgpu_mgr = kvzalloc(sizeof(*vgpu_mgr), GFP_KERNEL); + if (!vgpu_mgr) { + ret = -ENOMEM; + goto fail_alloc_vgpu_mgr; + } + + vgpu_mgr->handle = handle; + vgpu_mgr->handle.data.priv = vgpu_mgr; + + ret = nvidia_vgpu_mgr_attach_handle(&handle); + if (ret) + goto fail_attach_handle; + + kref_init(&vgpu_mgr->refcount); + mutex_init(&vgpu_mgr->vgpu_id_lock); + + mutex_unlock(&vgpu_mgr_attach_lock); + return vgpu_mgr; + +fail_attach_handle: + kvfree(vgpu_mgr); +fail_alloc_vgpu_mgr: + mutex_unlock(&vgpu_mgr_attach_lock); + vgpu_mgr = ERR_PTR(ret); + return vgpu_mgr; +} +EXPORT_SYMBOL(nvidia_vgpu_mgr_get); + +MODULE_LICENSE("Dual MIT/GPL"); +MODULE_AUTHOR("Zhi Wang "); +MODULE_DESCRIPTION("NVIDIA VGPU manager - core module to support VFIO PCI driver for NVIDIA vGPU"); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h new file mode 100644 index 000000000000..2efd96644098 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ +#ifndef __NVIDIA_VGPU_MGR_H__ +#define __NVIDIA_VGPU_MGR_H__ + +#include "nvkm.h" + +#define NVIDIA_MAX_VGPUS 2 + +struct nvidia_vgpu_info { + int id; + u32 gfid; + u32 dbdf; +}; + +struct nvidia_vgpu { + struct mutex lock; + atomic_t status; + struct pci_dev *pdev; + + u8 *vgpu_type; + struct nvidia_vgpu_info info; + struct nvidia_vgpu_mgr *vgpu_mgr; +}; + +struct nvidia_vgpu_mgr { + struct kref refcount; + struct nvidia_vgpu_mgr_handle handle; + + struct mutex vgpu_id_lock; + struct nvidia_vgpu *vgpus[NVIDIA_MAX_VGPUS]; + atomic_t num_vgpus; +}; + +struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev); +void nvidia_vgpu_mgr_put(struct nvidia_vgpu_mgr *vgpu_mgr); + +int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu); +int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type); + +#endif From patchwork Sun Sep 22 12:49:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809111 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2074.outbound.protection.outlook.com [40.107.92.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C674F4D5BD for ; Sun, 22 Sep 2024 12:50:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.74 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009458; cv=fail; b=A5yMBlIA5F7ZZc1lzDCTZVB8Mk8PkIard9ddlWXFX+AB1SuAT6njf1SSId+EyUPX0oBZBYYhIbMw9MrWW30BVo53slnp/ibhgPITWATtq4jd5bhbqvVP2a3YbsrduPovmWuV94uIV3t2SVa/C43u9iY8O52m8RqPcQUshKaV9yY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009458; c=relaxed/simple; bh=ff1xskTw26AfajV1se3vpJDotZgOnYUB9qLw8974YNo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=k/ypkl3ue9ok0Mu0LtHRzxeurZx0uumS0C7xwg3w3N6aNnnrXOE/PyAPdJWpwkhEPdUYx9vWyxvWg2FJfp9w7i0pFF5v+qDQB6LRwdbd5Rz+kF6oa7OkJEWPAMl3DPMkM+yAoXYG50XmGXn3l8w1/N5wXGjK1jUdtiBDflPVYac= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Fwsc9/HE; arc=fail smtp.client-ip=40.107.92.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Fwsc9/HE" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UY+7sb0Ht54/LdFdP4h4Kt1N4LOkJrrNCbJnhhYwOonSwzA7IRcNyzPzo5I69bsYAsOsJ2uep8dI7hIrPQMKVo1nwVX+PpzVlGlf3awxu1UCNcqt/v40ASMVOuEopNhJ/5eVvDnfayCJC56OMycdGWL1T5d5rJhUPkC9AnbIeUZ3tDQfTdCdyr1omMqyMlUODWMmFqblhL5/UK5puzThq2l6MaoWnTGyeIpAVF3FojpiIaFUEoZt0k0FtNTAs12FycWCp5l8bAvEErwtRUF3XEcw6LKOUmWh054zb6IIt8AWbU7H6E+SwY93W/ENPs8q1GVlG5slPi5iMbleE+cm3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NuMxwSlxxwMBgJ7m0PiWCabGj7yS2OaIHe1FmdY3dpc=; b=aQ0+aUZbIdaSi/j9y1sudRO6LoGgKvNlwwQOQH6vGfS1z8i/h3kdXgOt7/7JUv8G8X3nMOo4YVAQcRCh6JWYhri2BJh+dkajv8J8kzRrppriiUREvmVbhSGPD5xwphe5xlew7G+sncotxYjOlyFEZjLkY5Of8tTexLXIsQm3sMeK13ar7sqQSTqoiXuzgx4xDSEVUiiiNCGY+d2+BiPnKfS2k396dfAmCJHFUBSpZSdiI/bbszZ0S9T+JFRJXsuTXHGNXiUYEProsLmgfCq5OTyiMawtT8CqD3UN8prXOcOCsf2h1CGfz1Q8LW+KK3IHAz9LdCBDfbWRCdHhhmwCEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NuMxwSlxxwMBgJ7m0PiWCabGj7yS2OaIHe1FmdY3dpc=; b=Fwsc9/HE0fgHRUlDj2SEmmZRDI+EWzc/OPQqEYgID15u6OXYcwWJwwGhjnJbBLeiXhVPPV/47a7KYF+u55vM4GfROxvmW76XfSyKM7ujd1/UKho97kpQn/nM28ryn31heupS33uTady9WQXk6nGdO1anXS78ouiywDRUvtNUSXF74818VqLnIabfwXP6NyYZ8Ikt0ylQy10Vjf4WiLfiTCCDQiRJq3UcFsbyhTt5gdkES4rwSvU5pC4UTo41bQgtqkz2vJiCbYxMbRlOFvxcRblANQfPA7RrUP+3qLsE4m7Gi0eG+x1oxxG9lUmuQLXjKevQ1xyEkHl14Jy/jGYkcQ== Received: from MN2PR11CA0018.namprd11.prod.outlook.com (2603:10b6:208:23b::23) by BY1PR12MB8445.namprd12.prod.outlook.com (2603:10b6:a03:523::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:52 +0000 Received: from BL6PEPF00020E63.namprd04.prod.outlook.com (2603:10b6:208:23b:cafe::b8) by MN2PR11CA0018.outlook.office365.com (2603:10b6:208:23b::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.29 via Frontend Transport; Sun, 22 Sep 2024 12:50:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E63.mail.protection.outlook.com (10.167.249.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:51 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:33 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:32 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:32 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 20/29] vfio/vgpu_mgr: allocate GSP RM client for NVIDIA vGPU manager Date: Sun, 22 Sep 2024 05:49:42 -0700 Message-ID: <20240922124951.1946072-21-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E63:EE_|BY1PR12MB8445:EE_ X-MS-Office365-Filtering-Correlation-Id: ff35fbd3-cca5-4f16-a581-08dcdb0530dd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: Hw541q24QC7q0/OoUxcFai8D0goWfDFqeMJyGwBIU0CgzSl6gdO+hOkZqpGOunoSONgfiCEDePsXQH3aA3n3n6cKS65Iy7KJREIs+it9GdIwL3UJQY8DAMKOEj5xkPb1Z3RCvgq9nQMl9Mqmfql5MAAZGeW72gCEIn1cmtbpSx1dWKg1HBqSSS1ME5Ja8TKZbHVXm/Lxyh0QgL+m/5umkW/pCMQDI19Cht06b6B44wkmAwJcQv7IbvcT1SraRE4QAFPJUqm78s8d+rRxr54AaZke/gJ5Mjo9UMi7D3ANlwzqHsV5+ULGrdCHNeA9D1zWaafpbdYVUwHuUoVigSA4gOtqQ58673HO9v026ikW4TiOBFBHNWpblFT+cgN9jkHmxcnrJZsfk9yODtBbZL+thHInAU33G3asE5aFZxJupS3UdbFeOJUR9fRAmcDBDq5ipCt/QGXNAjHvZgnleE/rffCfZ+N8ZlKwAJJmkizcXsjwrxGzRznBb0al6v+F+34UU07XiH0r1Djhtmv1HbQP6JcbvD3CaiLIjJ39Ca2/YL8AdEOEpWzZ4hOihpHk6Uh5nOpob5bJQ+9nEoI0eFrqwzqBKidn19GLYBgEwzbfFFb3+t+yfVUBLN7QMLy8yKqfExUj6dLRZsMEXX62xA2pTabh3KTdRRkeCZTJv+1FecrNgJ0dTFsakVNTYW7+ev5ayfZSC1J/lrPVtM/dlcFB7zPF5oHVdpN9pczwH4wtiZEiKGFIEbcrkw5pHinjizVXG/b7vNjPOObfNFH/tR2yjLD7FF7dw10+6AAttN8NdkUZ+GpQe8Lg/uva31nJehrxX2OxFrKm+2PLNWvFRizbRNT4E9uFDo9dnoxYGnRyD8/vT8ZtF2SQFE4oeW9ct47ZFkcQcF+mjs1cNgU98EHJNKIzNnestZ8q9PEw9Th5UuY714tTffgs+xZFfg+vOvSeEnQfd+ikAhyDh1sAtuX+k/cDlSDNXKVHQ7iQx7vDY4XUaBgccjQBVKEpUM7gfJZTNU1o7pJDtXZxIQ82ReFsm5BiXzE/gKPtWmCznVE/V+rH0XyHTND9B7dHYogtHQpmRe1hPGF7wCR7iQMJX9El2MdT3HII8wFMTpQQC3icSC/Pb+1SbthysLiK2i8bROxfPz1JaqKD1358ZA6tui+FUYD7SNFE8Q/QQPcq5XNhTYMRMN2DrItlKZoAYfvFFTNoiFFuCXi4YPIG+WCmqyw43/sxK6sd8zDXhkqPQKJ4G6631WSlxrwYEOTg3Dhm2Qu/jwpN/YNFnfMT9M7w4lbo0D+gq/vbBWlZPLd/Qzg+n35zGkvaNs+E9xLy+Fibl7HOwTnpSLUA7HbR27XzPQrTgoaXFxDCv2m+01dCyQKHQKx8R7YfrvCUf8v2Lj9N2BAuF09dMBgTrG5wF9l8uVpoLVAx3gpRw6nQWEsBbOxRPOQNz3MUM6bPEF2/v6rCZLT5 X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:51.9044 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff35fbd3-cca5-4f16-a581-08dcdb0530dd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E63.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR12MB8445 A GSP RM client is required when talking to the GSP firmware via GSP RM controls. In order to create vGPUs, NVIDIA vGPU manager requires a GSP RM client to upload vGPU types to GSP. Allocate a dedicated GSP RM client for NVIDIA vGPU manager. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 9 +++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c | 8 ++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 2 ++ 3 files changed, 19 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 4c75431ee1f6..939f3b420bb3 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -43,4 +43,13 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_detach_handle(h) \ (h)->ops->detach_handle((h)->pf_drvdata) +#define nvidia_vgpu_mgr_alloc_gsp_client(m, c) \ + m->handle.ops->alloc_gsp_client(m->handle.pf_drvdata, c) + +#define nvidia_vgpu_mgr_free_gsp_client(m, c) \ + m->handle.ops->free_gsp_client(c) + +#define nvidia_vgpu_mgr_get_gsp_client_handle(m, c) \ + m->handle.ops->get_gsp_client_handle(c) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c index dc2a73f95650..812b7be00bee 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c @@ -12,6 +12,7 @@ static void vgpu_mgr_release(struct kref *kref) struct nvidia_vgpu_mgr *vgpu_mgr = container_of(kref, struct nvidia_vgpu_mgr, refcount); + nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu_mgr->gsp_client); nvidia_vgpu_mgr_detach_handle(&vgpu_mgr->handle); kvfree(vgpu_mgr); } @@ -82,9 +83,16 @@ struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev) kref_init(&vgpu_mgr->refcount); mutex_init(&vgpu_mgr->vgpu_id_lock); + ret = nvidia_vgpu_mgr_alloc_gsp_client(vgpu_mgr, + &vgpu_mgr->gsp_client); + if (ret) + goto fail_alloc_gsp_client; + mutex_unlock(&vgpu_mgr_attach_lock); return vgpu_mgr; +fail_alloc_gsp_client: + nvidia_vgpu_mgr_detach_handle(&vgpu_mgr->handle); fail_attach_handle: kvfree(vgpu_mgr); fail_alloc_vgpu_mgr: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index 2efd96644098..f4416e6ed8f9 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -32,6 +32,8 @@ struct nvidia_vgpu_mgr { struct mutex vgpu_id_lock; struct nvidia_vgpu *vgpus[NVIDIA_MAX_VGPUS]; atomic_t num_vgpus; + + struct nvidia_vgpu_gsp_client gsp_client; }; struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev); From patchwork Sun Sep 22 12:49:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809109 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2084.outbound.protection.outlook.com [40.107.101.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A868C33E1 for ; Sun, 22 Sep 2024 12:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.84 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009454; cv=fail; b=pH0loMNqKX8+cvcIrh5NWXWD4BFLA15qSQKu4NJTsN5LUdt3sHmSjXnLU/6wUWG4Xs+BSebDC3Gre5zFKOpdnQNROyavJg+ZGnBr8uI2jPs0xSFhi27c9qBYZu0iwWzFK3rHQsh/TOBJIleZ+wR/IHCKQTk+IArEG7dYu7yLjf8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009454; c=relaxed/simple; bh=0RFX1/ACE+MKYEwo5cye2OeLS4IkwZ0AWWQmoFZU0Qc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mE6C4AwRWPvPUQ/v9wxQXqLHTej7kHznaSht3LuMdKH78eywJ0E8CrxFjd9y0jF6zKUMf+8UuVW3Wgsxa45aNnbbq687axRagt+s/0X4T4wVQGmql8aqX62e5LRJt5IR0bnXMrTk9vhd7l4wcZRuNOS8Z/9cOJEX3gFWmia1Ehs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=SBjzbLrh; arc=fail smtp.client-ip=40.107.101.84 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="SBjzbLrh" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NDmEhOdkkOTCU6GFkblKW8S2U//CEe33VBbpIz3f2G+S5j4AmOZgFXWThsPMX1rj/8mbPo12S3wq6fLDRKJzZEUuze8AgBkjEPkv6obOBBGodV/aweeRCm6z3MMMNkAQSsXh8Q5FB6sWaUfc3NVKieevwjaPpnW+T9CysoQQyrri/MpK7Zg8zgWUinxChiD2aCsLwwFVh5Rd8MOKGiSbktn4Xf4tnKc8Gc64bQjTcEGhBYpRXXpto91gAOO0GfE761Vmmae/FHynLlZQOoZMixH8bGj7HnZ3IYY4JPMrtJGr6xV0MOPe70hmfoC6jQ1vQ3Ihv/P+pS8v1IseDcaSHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7SkNKN50x03QkRUlKLLsYAd9KfXmRZOGt+zBISb9yD0=; b=YteR9xdf/mo9BDrrrR6ewzXNLILKofLZJnmGsn4YxTe5ASlvi2cEwK4bOSoculwi8HUgHkwrOrqWRb76LA0RcSmbD7cNO+wg1oo0U6aPQJZv6BJh5QcOHvqG5nqoQEcUe5lpcYpY5WGs7mDWnxCL1UgeRmgtNB31p7prAVV9NW0HSZcPfISjJ6r+Z8EtqLxiJP4htL0tc7ze0XyVE16BBQ7voEVL/kKU/+OynB/hCy4gc+cepRkABG+sPUlzZtWcj1GI7L4jUrck8x3zfo2PboXKmhAfN9YK+aZwcPpAE2rRSSTODx/GAR6TuBXewvHqvflpMzuI94TOhvbU3irzvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7SkNKN50x03QkRUlKLLsYAd9KfXmRZOGt+zBISb9yD0=; b=SBjzbLrhTj+yvVCQ8uFtxAHn1XU8oWUgR6nXObB15IHf1YoRogyvpYj/9BkFi6tRP8MFivBSzkQ9xs4zoRdJMdILt7UtXmmgOKdB35zhQ7tugnDxiUDgzPXd0nY+sNeb+ybTv2sMA6pdhL+sqa53SZHYrtcwFMYK/vlfeC5u9bs8eXn9oykDrOg3N6+NH/7OL0+a27Zk+mBXdzVHCRJOeaQDimpvv664F5RG/jz4Lu3ed8ywqiKM1Pfgb/4d5MiHu5wHWsOn2EKkUELs0NjkcW8cwYfDc0Ijq3tlF8YdR8htOXiuAfpXL8bzkaO4S+WCJsQxZ7KA+gZi8UgJ1rFc3g== Received: from BY1P220CA0002.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::6) by SJ0PR12MB6879.namprd12.prod.outlook.com (2603:10b6:a03:484::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.22; Sun, 22 Sep 2024 12:50:45 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::c5) by BY1P220CA0002.outlook.office365.com (2603:10b6:a03:59d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24 via Frontend Transport; Sun, 22 Sep 2024 12:50:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:45 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:33 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:33 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:32 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 21/29] vfio/vgpu_mgr: introduce vGPU type uploading Date: Sun, 22 Sep 2024 05:49:43 -0700 Message-ID: <20240922124951.1946072-22-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|SJ0PR12MB6879:EE_ X-MS-Office365-Filtering-Correlation-Id: 9818ba14-6f65-482f-fa5e-08dcdb052cd3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?q?LmZ9K+5bdqcNp+VQJn3Hc9J4NUIzcs6?= =?utf-8?q?4yIhOfFs0LRzbO5jnKXNzD+/Tqrsf6Nonljy97wq+F4q2HV4qie7+qzZ4eOhTqsn3?= =?utf-8?q?jHc6SbhXgpaTFc2q8YhvcoiJIUKgOblt+tkKGNLZa3S+if+VKUF/WVyOsijqcaXwi?= =?utf-8?q?+bfFmD+AvhEBgqfcDRGEwfaterUNEZqvuguBv0aJglyVOB2Q/OFmD0J+pRlQZqT02?= =?utf-8?q?eyy8AFLecU5EdvuiqfvoShv+pH4I5FNpWKOzsgHNQp9p5euBhVpuls1AD5DOWl3pm?= =?utf-8?q?fU7gcXi2IBY8DC/yhps6fhiYl3aitgu3GY5Zlf8pt8nQ4D+bZQQfp9IZHMEEMvobK?= =?utf-8?q?tgEBmtvi59BwFbuslOcdV7Pq0tJQgu47ZpEe4YhzczhNQ1tUA2EWwc0RZLDeZxJST?= =?utf-8?q?zs4ghuwwNXeIa/qTgT4EeuWXvU/a82MP6R9OWaSOEV8Cm2jE9W9HvyNVpKX9dm+1q?= =?utf-8?q?nSG8pjkCHnMGBOz2n8gqo7qGU7s2I7pREIoUQJN8E4QqcWcdkgQ+SvrbeCDzOf7LG?= =?utf-8?q?hvbFurW4pF86033fTq/2g8kX+F+1+NLq3mqwYX0C3bEGGmtAoWorz82ndzW6JUy4C?= =?utf-8?q?69bi4Qx7QgbWpq0vKkLbNBJqaiGTx+iHxhwqd2XiKMgafyK5DBdkHuxfVNfgqO5zQ?= =?utf-8?q?oe0apphdkKA/jMX/N97DMmCHDpCjlkRP7O4aTwQ1x2syz3kRCWc7cWwbV5ZbqCZ/O?= =?utf-8?q?DaFVj/uz8gYtRVtc7UcgAWPSEiX1zdSXp8shj2Iv+OPnW1gw3xJbTe4AJcRoDzDp7?= =?utf-8?q?Prs9mGywIG1nZv4gTEGwIAnsPXqMaJUIJziUuDRIeSUPHuDdvaWuG2ytk2wCLrYKh?= =?utf-8?q?TO5fzKYgtYamATaTksuHHsIWNilyf4ClGmHF751bfhSfROogAZ5MjdgA03ofO5ALy?= =?utf-8?q?39ELib1154xWhNn1EuTFZMCd30zFCvfSBR9uMNsUlhL5wOxuKrkeV4MamjE17BfoI?= =?utf-8?q?+HuyoGdSDI1I4V7YExdlpZFleu140Ky4LD5pwvC55A/jDVe0quHiQVn7exOiGvg48?= =?utf-8?q?3coIr30DRprKiIx6O+AD7YL3AkY8Mc2dWbuarjMr+yocvK/+B22wgB6wBEMSRY2vv?= =?utf-8?q?dx3vYqAuK37fwb76ckYWbzZ8RBB32h8JAQeMiwuE5OxKzCe7f9N9C+QBSgCLIeKw7?= =?utf-8?q?/vPRyMwy3vNv+yHzYUNPfThqWA3ZVnZB1zvpB2sWVnso1RmiExm2/+gnJzzWngssK?= =?utf-8?q?czvkhul9/DT9/R928cNLfwWMcP16DZ0gZG4dR/7L2Hcv182TptToXrn2yhr8XQKO+?= =?utf-8?q?DeKNO/3c7LfJxzFFYJmwalMXBJADzdxh7iw2C8Co9gXyr64Qyhe4MlfQUAe3tkSkT?= =?utf-8?q?y81VbgNKgvacA0WLkCUSV/uZ+R1wwEPhSFn5UIgcQC5klkU+U/pXu6ez9wB0FmMnZ?= =?utf-8?q?o1xbHwOsj9p?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:45.2395 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9818ba14-6f65-482f-fa5e-08dcdb052cd3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6879 Each type of vGPU is designed to meet specific requirements, from supporting multiple users with demanding graphics applications to powering AI workloads in virtualized environments. To create a vGPU associated with a vGPU type, the vGPU type specs are required to be uploaded to GSP firmware. Intorduce vGPU type uploading defs and upload vGPU type specs when vGPU is enabled. Upload a default vGPU type: L40-24Q. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/Makefile | 4 +- .../nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h | 33 ++ .../ctrl/ctrl2080/ctrl2080vgpumgrinternal.h | 62 +++ .../common/sdk/nvidia/inc/ctrl/ctrla081.h | 109 ++++ .../pci/nvidia-vgpu/include/nvrm/nvtypes.h | 24 + drivers/vfio/pci/nvidia-vgpu/nvkm.h | 12 + drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c | 6 + drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 5 + drivers/vfio/pci/nvidia-vgpu/vgpu_types.c | 466 ++++++++++++++++++ 9 files changed, 720 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrla081.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/nvtypes.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/vgpu_types.c diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile index 1d2c0eb1fa5c..bd65fa548ea1 100644 --- a/drivers/vfio/pci/nvidia-vgpu/Makefile +++ b/drivers/vfio/pci/nvidia-vgpu/Makefile @@ -1,3 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only +ccflags-y += -I$(srctree)/$(src)/include + obj-$(CONFIG_NVIDIA_VGPU_MGR) += nvidia-vgpu-mgr.o -nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o +nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o vgpu_types.o diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h new file mode 100644 index 000000000000..5b6750f6f5a2 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080gpu.h @@ -0,0 +1,33 @@ +#ifndef __src_common_sdk_nvidia_inc_ctrl_ctrl2080_ctrl2080gpu_h__ +#define __src_common_sdk_nvidia_inc_ctrl_ctrl2080_ctrl2080gpu_h__ + +/* Excerpt of RM headers from https://github.com/NVIDIA/open-gpu-kernel-modules/tree/535.113.01 */ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2006-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +#define NV_GRID_LICENSE_INFO_MAX_LENGTH (128) + +#define NV2080_GPU_MAX_NAME_STRING_LENGTH (0x0000040U) + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h new file mode 100644 index 000000000000..f44cdc733229 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h @@ -0,0 +1,62 @@ +#ifndef __src_common_sdk_nvidia_inc_ctrl_ctrl2080_ctrl2080vgpumgrinternal_h__ +#define __src_common_sdk_nvidia_inc_ctrl_ctrl2080_ctrl2080vgpumgrinternal_h__ + +/* Excerpt of RM headers from https://github.com/NVIDIA/open-gpu-kernel-modules/tree/535.113.01 */ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2021-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +/* + * NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE + * + * This command is used to add a new vGPU config to the pGPU in physical RM. + * Unlike NVA081_CTRL_CMD_VGPU_CONFIG_SET_INFO, it does no validation + * and is only to be used internally. + * + * discardVgpuTypes [IN] + * This parameter specifies if existing vGPU configuration should be + * discarded for given pGPU + * + * vgpuInfoCount [IN] + * This parameter specifies the number of entries of virtual GPU type + * information + * + * vgpuInfo [IN] + * This parameter specifies virtual GPU type information + * + * Possible status values returned are: + * NV_OK + * NV_ERR_OBJECT_NOT_FOUND + * NV_ERR_NOT_SUPPORTED + */ +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE (0x20804003) /* finn: Evaluated from "(FINN_NV20_SUBDEVICE_0_VGPU_MGR_INTERNAL_INTERFACE_ID << 8) | NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS_MESSAGE_ID" */ + +#define NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS_MESSAGE_ID (0x3U) + +typedef struct NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS { + NvBool discardVgpuTypes; + NvU32 vgpuInfoCount; + NV_DECLARE_ALIGNED(NVA081_CTRL_VGPU_INFO vgpuInfo[NVA081_MAX_VGPU_TYPES_PER_PGPU], 8); +} NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS; + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrla081.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrla081.h new file mode 100644 index 000000000000..c40740e3f9df --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrla081.h @@ -0,0 +1,109 @@ +#ifndef __src_common_sdk_nvidia_inc_ctrl_ctrla081_h__ +#define __src_common_sdk_nvidia_inc_ctrl_ctrla081_h__ + +/* Excerpt of RM headers from https://github.com/NVIDIA/open-gpu-kernel-modules/tree/535.113.01 */ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2014-2020 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +#include + +#define NVA081_CTRL_VGPU_CONFIG_INVALID_TYPE 0x00 +#define NVA081_MAX_VGPU_TYPES_PER_PGPU 0x40 +#define NVA081_MAX_VGPU_PER_PGPU 32 +#define NVA081_VM_UUID_SIZE 16 +#define NVA081_VGPU_STRING_BUFFER_SIZE 32 +#define NVA081_VGPU_SIGNATURE_SIZE 128 +#define NVA081_VM_NAME_SIZE 128 +#define NVA081_PCI_CONFIG_SPACE_SIZE 0x100 +#define NVA081_PGPU_METADATA_STRING_SIZE 256 +#define NVA081_EXTRA_PARAMETERS_SIZE 1024 + +/* + * NVA081_CTRL_VGPU_CONFIG_INFO + * + * This structure represents the per vGPU information + * + */ +typedef struct NVA081_CTRL_VGPU_INFO { + // This structure should be in sync with NVA082_CTRL_CMD_HOST_VGPU_DEVICE_GET_VGPU_TYPE_INFO_PARAMS + NvU32 vgpuType; + NvU8 vgpuName[NVA081_VGPU_STRING_BUFFER_SIZE]; + NvU8 vgpuClass[NVA081_VGPU_STRING_BUFFER_SIZE]; + NvU8 vgpuSignature[NVA081_VGPU_SIGNATURE_SIZE]; + NvU8 license[NV_GRID_LICENSE_INFO_MAX_LENGTH]; + NvU32 maxInstance; + NvU32 numHeads; + NvU32 maxResolutionX; + NvU32 maxResolutionY; + NvU32 maxPixels; + NvU32 frlConfig; + NvU32 cudaEnabled; + NvU32 eccSupported; + NvU32 gpuInstanceSize; + NvU32 multiVgpuSupported; + NV_DECLARE_ALIGNED(NvU64 vdevId, 8); + NV_DECLARE_ALIGNED(NvU64 pdevId, 8); + NV_DECLARE_ALIGNED(NvU64 profileSize, 8); + NV_DECLARE_ALIGNED(NvU64 fbLength, 8); + NV_DECLARE_ALIGNED(NvU64 gspHeapSize, 8); + NV_DECLARE_ALIGNED(NvU64 fbReservation, 8); + NV_DECLARE_ALIGNED(NvU64 mappableVideoSize, 8); + NvU32 encoderCapacity; + NV_DECLARE_ALIGNED(NvU64 bar1Length, 8); + NvU32 frlEnable; + NvU8 adapterName[NV2080_GPU_MAX_NAME_STRING_LENGTH]; + NvU16 adapterName_Unicode[NV2080_GPU_MAX_NAME_STRING_LENGTH]; + NvU8 shortGpuNameString[NV2080_GPU_MAX_NAME_STRING_LENGTH]; + NvU8 licensedProductName[NV_GRID_LICENSE_INFO_MAX_LENGTH]; + NvU32 vgpuExtraParams[NVA081_EXTRA_PARAMETERS_SIZE]; + NvU32 ftraceEnable; + NvU32 gpuDirectSupported; + NvU32 nvlinkP2PSupported; + NvU32 multiVgpuExclusive; + NvU32 exclusiveType; + NvU32 exclusiveSize; + // used only by NVML + NvU32 gpuInstanceProfileId; +} NVA081_CTRL_VGPU_INFO; + +/* + * NVA081_CTRL_VGPU_CONFIG_INFO_PARAMS + * + * This structure represents the vGPU configuration information + * + */ +#define NVA081_CTRL_VGPU_CONFIG_INFO_PARAMS_MESSAGE_ID (0x1U) + +typedef struct NVA081_CTRL_VGPU_CONFIG_INFO_PARAMS { + NvBool discardVgpuTypes; + NV_DECLARE_ALIGNED(NVA081_CTRL_VGPU_INFO vgpuInfo, 8); + NvU32 vgpuConfigState; +} NVA081_CTRL_VGPU_CONFIG_INFO_PARAMS; + +/* VGPU Config state values */ +#define NVA081_CTRL_VGPU_CONFIG_STATE_UNINITIALIZED 0 +#define NVA081_CTRL_VGPU_CONFIG_STATE_IN_PROGRESS 1 +#define NVA081_CTRL_VGPU_CONFIG_STATE_READY 2 + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/nvtypes.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/nvtypes.h new file mode 100644 index 000000000000..e6833df1ccc7 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/nvtypes.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: MIT */ +#ifndef __NVRM_NVTYPES_H__ +#define __NVRM_NVTYPES_H__ + +#define NV_ALIGN_BYTES(a) __attribute__ ((__aligned__(a))) +#define NV_DECLARE_ALIGNED(f,a) f __attribute__ ((__aligned__(a))) + +typedef u32 NvV32; + +typedef u8 NvU8; +typedef u16 NvU16; +typedef u32 NvU32; +typedef u64 NvU64; + +typedef void* NvP64; + +typedef NvU8 NvBool; +typedef NvU32 NvHandle; +typedef NvU64 NvLength; + +typedef NvU64 RmPhysAddr; + +typedef NvU32 NV_STATUS; +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 939f3b420bb3..065bb7aa55f8 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -52,4 +52,16 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_get_gsp_client_handle(m, c) \ m->handle.ops->get_gsp_client_handle(c) +#define nvidia_vgpu_mgr_rm_ctrl_get(m, g, c, s) \ + m->handle.ops->rm_ctrl_get(g, c, s) + +#define nvidia_vgpu_mgr_rm_ctrl_wr(m, g, c) \ + m->handle.ops->rm_ctrl_wr(g, c) + +#define nvidia_vgpu_mgr_rm_ctrl_rd(m, g, c, s) \ + m->handle.ops->rm_ctrl_rd(g, c, s) + +#define nvidia_vgpu_mgr_rm_ctrl_done(m, g, c) \ + m->handle.ops->rm_ctrl_done(, c) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c index 812b7be00bee..dcb314b14f91 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c @@ -88,9 +88,15 @@ struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev) if (ret) goto fail_alloc_gsp_client; + ret = nvidia_vgpu_mgr_init_vgpu_types(vgpu_mgr); + if (ret) + goto fail_init_vgpu_types; + mutex_unlock(&vgpu_mgr_attach_lock); return vgpu_mgr; +fail_init_vgpu_types: + nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu_mgr->gsp_client); fail_alloc_gsp_client: nvidia_vgpu_mgr_detach_handle(&vgpu_mgr->handle); fail_attach_handle: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index f4416e6ed8f9..eb2df9f0fe07 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -33,6 +33,9 @@ struct nvidia_vgpu_mgr { struct nvidia_vgpu *vgpus[NVIDIA_MAX_VGPUS]; atomic_t num_vgpus; + u8 **vgpu_types; + u32 num_vgpu_types; + struct nvidia_vgpu_gsp_client gsp_client; }; @@ -42,4 +45,6 @@ void nvidia_vgpu_mgr_put(struct nvidia_vgpu_mgr *vgpu_mgr); int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu); int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type); +int nvidia_vgpu_mgr_init_vgpu_types(struct nvidia_vgpu_mgr *vgpu_mgr); + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_types.c b/drivers/vfio/pci/nvidia-vgpu/vgpu_types.c new file mode 100644 index 000000000000..d402663e01bc --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_types.c @@ -0,0 +1,466 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include + +#include +#include +#include + +#include "vgpu_mgr.h" + +unsigned char vgpu_type_869[] = { + 0x65, 0x03, 0x00, 0x00, 0x4e, 0x56, 0x49, 0x44, 0x49, 0x41, 0x20, 0x4c, + 0x34, 0x30, 0x2d, 0x31, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x4e, 0x56, 0x53, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x6c, 0xab, 0x5b, + 0x8f, 0xdb, 0xd5, 0x37, 0x09, 0x9a, 0x35, 0x04, 0x91, 0xde, 0x94, 0xb3, + 0x63, 0x78, 0xff, 0xca, 0x38, 0x2f, 0x0a, 0xee, 0xe2, 0x76, 0xb0, 0xe9, + 0xda, 0x9d, 0xf2, 0x5c, 0x2f, 0xb2, 0xe3, 0xe8, 0xfb, 0x96, 0x64, 0x9f, + 0x5a, 0x59, 0x5f, 0x1e, 0x13, 0x19, 0x5a, 0x45, 0x3a, 0xf4, 0x85, 0x04, + 0xba, 0xc5, 0x53, 0x1c, 0xfc, 0x3a, 0x8b, 0x78, 0x98, 0x36, 0x49, 0x66, + 0x91, 0x0d, 0x99, 0x55, 0x59, 0xec, 0x8d, 0xd7, 0x13, 0x4a, 0x0a, 0x60, + 0x2b, 0xd1, 0x96, 0x05, 0x13, 0x92, 0xde, 0xe0, 0xf8, 0x83, 0xac, 0x0b, + 0xf7, 0x3c, 0xb5, 0x35, 0x44, 0x90, 0xcb, 0x6b, 0x63, 0xba, 0xc8, 0xaa, + 0x3c, 0x34, 0xfd, 0xf6, 0x94, 0xdd, 0x74, 0x85, 0x3e, 0x44, 0xb5, 0x4d, + 0xff, 0x99, 0x40, 0x52, 0x8c, 0xd2, 0xec, 0x99, 0xc5, 0x77, 0x79, 0x30, + 0x8d, 0xb4, 0xba, 0x70, 0x47, 0x52, 0x49, 0x44, 0x2d, 0x56, 0x69, 0x72, + 0x74, 0x75, 0x61, 0x6c, 0x2d, 0x50, 0x43, 0x2c, 0x32, 0x2e, 0x30, 0x3b, + 0x51, 0x75, 0x61, 0x64, 0x72, 0x6f, 0x2d, 0x56, 0x69, 0x72, 0x74, 0x75, + 0x61, 0x6c, 0x2d, 0x44, 0x57, 0x53, 0x2c, 0x35, 0x2e, 0x30, 0x3b, 0x47, + 0x52, 0x49, 0x44, 0x2d, 0x56, 0x69, 0x72, 0x74, 0x75, 0x61, 0x6c, 0x2d, + 0x57, 0x53, 0x2c, 0x32, 0x2e, 0x30, 0x3b, 0x47, 0x52, 0x49, 0x44, 0x2d, + 0x56, 0x69, 0x72, 0x74, 0x75, 0x61, 0x6c, 0x2d, 0x57, 0x53, 0x2d, 0x45, + 0x78, 0x74, 0x2c, 0x32, 0x2e, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x20, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, + 0x40, 0x0b, 0x00, 0x00, 0x00, 0x00, 0xfa, 0x00, 0x2d, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x6d, 0x17, 0xb5, 0x26, + 0x00, 0x00, 0x00, 0x00, 0xb5, 0x26, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 +}; + +static unsigned char *vgpu_types[] = { + vgpu_type_869, +}; + +/** + * nvidia_vgpu_mgr_init_vgpu_types - Upload the default vGPU types. + * @vgpu_mgr: NVIDIA vGPU manager + * + * Returns: 0 on success, others on failure. + */ +int nvidia_vgpu_mgr_init_vgpu_types(struct nvidia_vgpu_mgr *vgpu_mgr) +{ + NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS *ctrl; + int i, ret; + + ctrl = nvidia_vgpu_mgr_rm_ctrl_get(vgpu_mgr, &vgpu_mgr->gsp_client, + NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE, + sizeof(*ctrl)); + if (IS_ERR(ctrl)) + return PTR_ERR(ctrl); + + ctrl->discardVgpuTypes = true; + ctrl->vgpuInfoCount = ARRAY_SIZE(vgpu_types); + + for (i = 0; i < ARRAY_SIZE(vgpu_types); i++) + memcpy(&ctrl->vgpuInfo[i], vgpu_types[i], sizeof(NVA081_CTRL_VGPU_INFO)); + + ret = nvidia_vgpu_mgr_rm_ctrl_wr(vgpu_mgr, &vgpu_mgr->gsp_client, + ctrl); + if (ret) + return ret; + + vgpu_mgr->vgpu_types = vgpu_types; + vgpu_mgr->num_vgpu_types = ARRAY_SIZE(vgpu_types); + + return 0; +} From patchwork Sun Sep 22 12:49:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809105 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2058.outbound.protection.outlook.com [40.107.93.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C4812CCC2 for ; Sun, 22 Sep 2024 12:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; cv=fail; b=Ad4KOqgGKbvqTk2jpjfCYbcHebTsTD17H182mUZZ+HjUroeGtq2i/ve6HywZjyxMwGKJY4Dwg+Pm+uwKxbcUBfgCEywyvzX41q62SoY1V3BY7Y6Tx4DF1+0kpdZpufRseqm9evAsvDNPOIVrD/GAWG7HaqDtjcsPfswOhH+ZyWo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; c=relaxed/simple; bh=aOWt9elKSOoR1bJCvJ5C3PYz1TkBxBXofeR5Yvrl+Ww=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eC3lpIqDWjDiSc4S51bKAWWgixQLSiQSjnKIBvwqDWvL6zXXv48+shodcCjn9w28CLj90udxvnL3iN1e14ymwsqKAdxAggPXiaAZLlDd1YPVykRP43UcrQDfgc1jd4tEDwXU1YPC9Jeu0Z8ysaS9tBj8rHGvoqkGG7svl4PvaW4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=CgIHs6xD; arc=fail smtp.client-ip=40.107.93.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="CgIHs6xD" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=YASe/G7rmiAp8cI5yxT2t5UIHRGIpYiClx/ZV2R5wBRqnNYzf8Wh1mHMbNYZpcLn2rH/qEEeAD9ZWMAQ62celTK4OqkRJJjY9IHwPvk2p1+ObjUfJEc5/McL7RyZdOCSYI5AneNZZaIpossySSJqyiR0F0AzTb4VORJ0JjHeP3ma8lm1Mx1boNNtXVpopk0qr1iWuqVcwCpuD1QAdX/Qnb21w3BOjSPuP2932Pt4xAMy1kTjpx4kIggaeF/0hpFPaZ/dqu0u1B2fP761BgrwkxTc9naYe1SKUeqDOCuJcCw9D+g3Wdb81qndNzf6otO7oF7Gd7qDAzPGS6y376PClw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ov5LC7GbiOMPGtzA4WRPVETEec9xsRePWztyrkfU+jo=; b=dhe6VlXn2zLRmAVTnFOdTWTu/dfDKgYryqFW8JqM//QtymnRm3ejPQaaLWdxNoWmvjfdLIYg+ssRfvJJJ6Qh6Pz/GkIESdEKo8CN+0/iY1aAeVwe3AEjF4u/fhsfoMS5Bud1t/iE/iW8q+X8Tvk9mRtQIPw1R4R04rAQrOF1OQtB8ZLlKQrYQl4u9JDtgm2snHtxWu1mo+Xgy14Jqnap7VGDzHJ6Bms+VFrwrg6usbSFPKqC6pgHETjeif7DYA08U87QjdllUeEjIJdHKOsvbTjaw+N/Y2mgubLF5HbdnasC71I3zoKdEQu8TTBTzJWcOEWfe+4VJMQ7amEW8M/zuQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ov5LC7GbiOMPGtzA4WRPVETEec9xsRePWztyrkfU+jo=; b=CgIHs6xDtCix6XNcH4x2RrbxQ/T1oDyfIUg+msy7Ly3BJ91v0VZGkf0mXTnkm1X3H3gl+qyTiSng7ZsGVomrbvbjATbNbzkf+3H/WJAHX+oQcdN2KXxmPID2Oqjqb+1VpgbIWCybby1vxskxg2d/O1p5Og/2jOmyV+Fg/pvPgHQZz55rIOiN9yjQJDA90C0Z7czwS1+TxeYOQvtEF5hJniwV/lOjyRhQD5WGq+d9GTJpjux55szFLaurgRsPxGNILJhAQCIXGmx8LQY0DCvzSuzfax02g15e042FvRFmWuneWQU9tW1m2n7og8s7B9zSV2beu5+/D0Y7SiXcFsFNbQ== Received: from BY1P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::17) by CY5PR12MB6203.namprd12.prod.outlook.com (2603:10b6:930:24::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:46 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::9) by BY1P220CA0013.outlook.office365.com (2603:10b6:a03:59d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:46 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:34 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:33 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:33 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 22/29] vfio/vgpu_mgr: allocate vGPU FB memory when creating vGPUs Date: Sun, 22 Sep 2024 05:49:44 -0700 Message-ID: <20240922124951.1946072-23-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|CY5PR12MB6203:EE_ X-MS-Office365-Filtering-Correlation-Id: 8608cda4-3fce-412e-2d1a-08dcdb052d89 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?kDP5HGo8XFqmr+rLex/pxjwmVbQqA5c?= =?utf-8?q?ej7HUgkhkJvK3bvOY5RHIls6RID7EnZWGDtnMwbutz9dWQwJuEG7+n3/yXByY1/+0?= =?utf-8?q?u4JAXbCoppuEgePNQr7g7KY7wL2z9/9vxq/pSqWSPsiQCVTk1yIbPNTYqbPknbP8z?= =?utf-8?q?UdyKvYOvQYxn4IZ0v5eROfTmrtiN6EdQbDU2fOc2HZhKKAWTaO+1l2ltADUGVXVnX?= =?utf-8?q?+YwG6tdzeyQyQJIq4o/otUdWgHxQxOKgShxcHQjfSqpU7/3Kc6U0zdGEHQecPrq1P?= =?utf-8?q?To59pUVnhKEDrWFs6G+alWVR4auxsw9xRlGSVFmYLJJ5VgZWKz9lvDkHsG8uGp9Gw?= =?utf-8?q?36dyYRuczfE8uP935EXcennyu2lvx5x7+qcfdTsRigwdKd4Zgdk/dYRkwGXVGJorp?= =?utf-8?q?GPolz2nl0cZhEo89v2HZ5EIDbz2M6eIoFj0rdarS1Fe6wEmGKKigYi52VHwqEU8rH?= =?utf-8?q?AlidDFadDc5cWMBiuV1jtMZ+k0uwnHyRs8jgWpAHB4snjHeSaYuG7Ipx4LOpvT5Ha?= =?utf-8?q?mgrfjrWgD9aMcER90k40z/Dd2Cu21RFFO36btq7IWpJKQSAbSmgaon1OoY5/j8Nij?= =?utf-8?q?530wwq8C9HK4Wk0BnfPYdR8zsWbvrQHBFXUaLzDFUbnIzbRta7et92LSXzk1OWBJC?= =?utf-8?q?uU8tTklV0gdoT5eZyuxf4k8+rebWZPNfcOkDyr2ufppM30xN/uR3/UULKOZmhd27x?= =?utf-8?q?fuFqmnN+UAsSV8AHLZzncQ1DR9Jnz8cd0yUGwPXMTe9BQCsCNGQOIJhXa8glsYRJm?= =?utf-8?q?R0uYRnoAxTOZ/2fXGBG8hQsbJn3WuoaTFBqEVbrFztQLlqljKsNKGk3vYpWO/ei+A?= =?utf-8?q?28xuxSyYfcSf5/c6LjiJs2k2LX12QofpciLQG9lxDcc1NuWW74B2aFoB8m6byEmir?= =?utf-8?q?eNVZwiqFY4CQ9y0ybUokAZRZXy6jsCF0KNtDqnJTN7UXHbYLz20VzkBhpVIvQh9/d?= =?utf-8?q?Kzw2ZqOrwpcacbTwU7Ii0ePNgbFcZWNzP178ybRzlGhrDu6tBcK7/oXVkl40vi0Hi?= =?utf-8?q?cILJ01ZQ1+azNfTdEsM6dIDeLLNlwjcDSpoHgabKl9KB/VzqdBe+OXeOa/+kzvRBM?= =?utf-8?q?g5C7m6PNDGKCb7soBA/shnq2I89mIXZEpDf3ULL5hIExdZR8/T7nM4laiXg5PjcQP?= =?utf-8?q?NP8GrGUl6EkoYhZIsj+++sFrQqMTLNRMqzK164HrrBlZ/sKKKnZa3lrHKBZp4ko3z?= =?utf-8?q?wP08Ev+YdyfN2If0ukH3eEz+vqFokC5Ue8zvXCf1p24prUNJWUzsCEfXAO2j0T+xU?= =?utf-8?q?4opYW9s2aC1tUud+PbXs17ooawRqET/bMriQZJFnT1M95p+rxoss5vcbwYBAHLoZA?= =?utf-8?q?HW3MtEo+YXi5s3xVR6ZfoURxVwm4fftLdlPx5UisaU9vwmJTTpIXqSZso2bOwSofJ?= =?utf-8?q?zsgXfe35qrz?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:46.4270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8608cda4-3fce-412e-2d1a-08dcdb052d89 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6203 Creating a vGPU requires allocating a portion of the FB memory from the NVKM. The size of the FB memory that a vGPU requires is from the vGPU type. Acquire the size of the required FB memory from the vGPU type. Allocate the FB memory from NVKM when creating a vGPU. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 6 ++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 38 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 2 ++ 3 files changed, 46 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 065bb7aa55f8..d3c77d26c734 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -64,4 +64,10 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_rm_ctrl_done(m, g, c) \ m->handle.ops->rm_ctrl_done(, c) +#define nvidia_vgpu_mgr_alloc_fbmem_heap(m, s) \ + m->handle.ops->alloc_fbmem(m->handle.pf_drvdata, s, true) + +#define nvidia_vgpu_mgr_free_fbmem_heap(m, h) \ + m->handle.ops->free_fbmem(h) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 34f6adb9dfe4..54e27823820e 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -3,6 +3,11 @@ * Copyright © 2024 NVIDIA Corporation */ +#include + +#include +#include + #include "vgpu_mgr.h" static void unregister_vgpu(struct nvidia_vgpu *vgpu) @@ -34,6 +39,29 @@ static int register_vgpu(struct nvidia_vgpu *vgpu) return 0; } +static void clean_fbmem_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + + nvidia_vgpu_mgr_free_fbmem_heap(vgpu_mgr, vgpu->fbmem_heap); + vgpu->fbmem_heap = NULL; +} + +static int setup_fbmem_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + NVA081_CTRL_VGPU_INFO *info = + (NVA081_CTRL_VGPU_INFO *)vgpu->vgpu_type; + struct nvidia_vgpu_mem *mem; + + mem = nvidia_vgpu_mgr_alloc_fbmem_heap(vgpu_mgr, info->fbLength); + if (IS_ERR(mem)) + return PTR_ERR(mem); + + vgpu->fbmem_heap = mem; + return 0; +} + /** * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance * @vgpu: the vGPU instance going to be destroyed. @@ -45,6 +73,7 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + clean_fbmem_heap(vgpu); unregister_vgpu(vgpu); return 0; } @@ -76,8 +105,17 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) return ret; + ret = setup_fbmem_heap(vgpu); + if (ret) + goto err_setup_fbmem_heap; + atomic_set(&vgpu->status, 1); return 0; + +err_setup_fbmem_heap: + unregister_vgpu(vgpu); + + return ret; } EXPORT_SYMBOL(nvidia_vgpu_mgr_create_vgpu); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index eb2df9f0fe07..35312d814996 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -23,6 +23,8 @@ struct nvidia_vgpu { u8 *vgpu_type; struct nvidia_vgpu_info info; struct nvidia_vgpu_mgr *vgpu_mgr; + + struct nvidia_vgpu_mem *fbmem_heap; }; struct nvidia_vgpu_mgr { From patchwork Sun Sep 22 12:49:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809113 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2084.outbound.protection.outlook.com [40.107.237.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EA714F602 for ; Sun, 22 Sep 2024 12:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.84 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009461; cv=fail; b=UgcI6uhHSbaycV0TkvN4Z1n2ybTyTrLS4lOcuAdCsWw7dtjofXFWcMtkasaJCemE9xtKJA9xRCBI55cSBx4hsuteTVKPCGI/gs8EOp9pqbRQvPn4aX2W6BhpFuMzTxbgZZ5MURayHeWegr9l1o7sCEaLggLNFlIO2JSqBp7k9G0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009461; c=relaxed/simple; bh=3ur1qtjgvQW32sthGgO0x9NgsbMm2aQsXjUqPHHaFI0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hiJ8NZYivEjyNPVTTN2v4EX2N8HJ3f/o/DaLW6HANQaXzvez1E4EMxoXcEOTbd/XyBVbw2RElSrnZkv3Obvq8oaYrFmA938NxgETqUBqoAlTl3DylJtEe76+a9LTClGXTktaSYTDji6VRIdVyT3pm/6me725JWDWeqxP69hzzzw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=CYs0LaXs; arc=fail smtp.client-ip=40.107.237.84 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="CYs0LaXs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ut/iK5G1+l/oYPVN7NLjhaZ1t8mhQyE++UyhdaU/NeqjxhWcfYD6DaBIeiKtc199NYcAdJr6YmDfgMOkmg+9JCDkdpohKE4rl1kBgeAFHqk4G0hiihUe0+bNvNd4e95pJ1eh/VXJQ8JGdtFk2murSpQcB0+nlN7EOu7ToLq7Styklgu1c9bFJ2IOqMW46bY5ARuFjz4pmlJcYXPSgcCcD7XFRGrSLAOi5mTymmZq6Ai99sPZknI6AGK5o6VXGAXBd6BmD1mVUb1LeP07c636qr+Qs6ktWLpE95MkxDN8+tyRFX/IpAzuaAuHuUOXtq9UG7KwD17vz/QjRyVE8oRDRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OUpmMuBusD75UIZyyB43ONShs/8YwJi0aW34Di9h4lY=; b=jf+7Whsm3xz6fVjt6sD8E0NNE5WbdGdj88UXZ7+Et0TEa9nt2CIavCP9ERZTizJtvyp+62DHofhLW3lu1gEOQqp88mbv6K088DdLUvsxTpTHu5I9GMly/+7gZRQ94FbjDwAVZvqIiNMqyt5nqC9Xrc8GMHHz3TES/VE5VLPPUINttzDMf7ZXVbqcHZ0TMy7lZjvQjMHjI2Ewx5p/990lYQG0ezIpTFCATfRVWvfDEz2cnKUFQZCW9etiqrdzTmPM/l/hjekcnWf+zJ+3mJzK7DgakkIBWxBCKiTAqKcXPoMe1yiVowGy49hAkb33evPBx0r4sG4OvZbJx0Hkn18AUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OUpmMuBusD75UIZyyB43ONShs/8YwJi0aW34Di9h4lY=; b=CYs0LaXsyqZYlkhZTjD4Q4kLfqkV65Ct66K6qLwFBTu6HM7gX539XTNabj40XtS5XzD3UlVxpO3ZUuE12JCA5z5Js57cW8QDZr+uiH8aBJZbguxWjYDcopvUV52qddI92zKqJvOEWmW4oCXpAIIBpoqXQPAMtTFnra2l18rJBSnwS2yKsq+5/uniFBrwWowkUpBp9i9hsip+xF2US8O5oAw2Y3wdG5Czdi4cVv6cRZoK9TrkXT8+UGI3JPRDRb1R2eHOXyRZf5MHFZN01vVEreYYBGlmRoSg1Dh2synuwuIg5DJ0YCRQ62bu5LronqFfTdm8yCxHB7Na1x7QBSxPhA== Received: from BN0PR04CA0138.namprd04.prod.outlook.com (2603:10b6:408:ed::23) by DS7PR12MB6285.namprd12.prod.outlook.com (2603:10b6:8:96::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:55 +0000 Received: from BL6PEPF00020E62.namprd04.prod.outlook.com (2603:10b6:408:ed:cafe::55) by BN0PR04CA0138.outlook.office365.com (2603:10b6:408:ed::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.27 via Frontend Transport; Sun, 22 Sep 2024 12:50:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E62.mail.protection.outlook.com (10.167.249.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:54 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:34 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:34 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:34 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 23/29] vfio/vgpu_mgr: allocate vGPU channels when creating vGPUs Date: Sun, 22 Sep 2024 05:49:45 -0700 Message-ID: <20240922124951.1946072-24-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E62:EE_|DS7PR12MB6285:EE_ X-MS-Office365-Filtering-Correlation-Id: cf5d345e-a600-4fd3-63c7-08dcdb0532a6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: 6PoPsy3NVn0wGMf9Q83167XlrQcMkL2W8JrtoXx0q2sG1PaIF50cmgR3fzQtqXxoPDyvFb7vqZdMIN6Do03IOl9B5rrJXyN4AiTRLg0X7o9jnaivwaXmr5Yf4af4xzeHsPyeCFL6fwRi2kUEOz3XksvMUdQFtOxyrPYEvdMW2G4ugtoDQi0LjoZdKISmhzEm4zNRwBvum0b4rymM3ifEhoG/4qqgLLypb7kIh7jBdleUILILDJYIBspNLzobUI4kU5gqUYAppQrSZjmMtEXg97KqqkcGOQElxA0fJdTxms+Fj4nQ6rChKDE3lk3+Th/QNqJeyCTYAvddgQ2/7y8zfjRf4dNtXWRS5jsdF1aiYii0MruQCJsFIAq+c95dZ1uICeRCUwCT5I+sUQXVIxpbk6ID0/AFc5pvYgBRagAOfyHZhGLHbn2tYXAj5YdUwiWBmvR22ipbLhrAFM6kiuPJrmLEQPxnSHXoBw0OxL4RxgCWMFS4/9Z2pRkrXLfeSZamTK9ffYW0PCajbTMo7nmq8lLVdvs4PX0IUtvNdo/pu8ecCeL88viNKqd7lFMfw5qSMVOJWM5jvDMreQv/UNWm+NjpVYQZOZkKCYGGhc39mYbbGwjA1n3bKCOryc1Ax3H8B8shiLL77WrLBI/TQtivsHI5fOgCyWGsEYS3rKpPlj/aJAvR0pcdVEjG/Gz4bjIb9PYbL9lr4mEHxwXsJbMgaTMcpa6ZaYD9MzVEMui6zqFktnVJPfru2J2SH7Qsttl/EsEvJE+eASBxqMrdCkdWUoX40ZktZxVUrI82lfEMvGDDxOHyVZTm9P+al9lUsPXlbkPcTh1AHO17mEmV4zmhjYv9ARqadV05Zg+4amZ8TiXm5T+H1RiICOPIHrQ5PrBlVSJKrMLFoJo7SG0TqM9Uo08mlvi3L9m52yVKC1M2dNp67RGr5Y/wIscXJk0aEl+sUhQSZOWHqLKjCXaCW/Mhu18qCgpr3RUIyIBAGMB/JYTFFjPHoAMAE0XNsxCcldix9ZSoFJHDPdxrMk5wDtJnIOQIvX98LQNgzi3A7vOvnkurGVs3EfYjh8nLiJ0IJI1Pg3wdPge9cRH4LVNcVYBXup6ysgjHZHps2dRzbZaRB1hero4HHQX0rvgDhWP0oKUZIt3k4wr9mIqs0gRr7EHPM2fNa/aJTqzrS8+ljsmTze0LEHrYiReZfvF3ZlHZpcjao/BvsnQhcIsanSF5eWaodqGg/9KF4bEloQ1ToIiIRY3x4Ztp1Qy1orJ5PMeMjBtIWNF6s5k5Z7TBX2ZOTvE0QxHh9CpuEEWuZPGa2AP10XipBvZmZkoA2M2u1T0aqm+1kwCts0hYcYnSDVnLRKhmnR4CAXgL2EaByrQoGgyffm1AU4GSc1fYJ2eUym3S3zeT6WrVqcgPl9bk8+J5H5nR2QJAOtlj1RXrkXv48emuRYf5w1bwSdOGVdh9KrR8UGUk X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:54.9290 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cf5d345e-a600-4fd3-63c7-08dcdb0532a6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E62.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6285 Creating a vGPU requires allocating a portion of the channels from the reserved channel pool. Allocate the channels from the reserved channel pool when creating a vGPU. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 6 +++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 32 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 7 ++++++ 3 files changed, 45 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index d3c77d26c734..b95b48edeb03 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -70,4 +70,10 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_free_fbmem_heap(m, h) \ m->handle.ops->free_fbmem(h) +#define nvidia_vgpu_mgr_alloc_chids(m, s) \ + m->handle.ops->alloc_chids(m->handle.pf_drvdata, s) + +#define nvidia_vgpu_mgr_free_chids(m, o, s) \ + m->handle.ops->free_chids(m->handle.pf_drvdata, o, s) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 54e27823820e..c48c22d8fbb4 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -62,6 +62,31 @@ static int setup_fbmem_heap(struct nvidia_vgpu *vgpu) return 0; } +static void clean_chids(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + struct nvidia_vgpu_chid *chid = &vgpu->chid; + + nvidia_vgpu_mgr_free_chids(vgpu_mgr, chid->chid_offset, chid->num_chid); +} + +static int setup_chids(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + struct nvidia_vgpu_chid *chid = &vgpu->chid; + int ret; + + ret = nvidia_vgpu_mgr_alloc_chids(vgpu_mgr, 512); + if (ret < 0) + return ret; + + chid->chid_offset = ret; + chid->num_chid = 512; + chid->num_plugin_channels = 0; + + return 0; +} + /** * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance * @vgpu: the vGPU instance going to be destroyed. @@ -73,6 +98,7 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + clean_chids(vgpu); clean_fbmem_heap(vgpu); unregister_vgpu(vgpu); return 0; @@ -109,10 +135,16 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) goto err_setup_fbmem_heap; + ret = setup_chids(vgpu); + if (ret) + goto err_setup_chids; + atomic_set(&vgpu->status, 1); return 0; +err_setup_chids: + clean_fbmem_heap(vgpu); err_setup_fbmem_heap: unregister_vgpu(vgpu); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index 35312d814996..0918823fdde7 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -15,6 +15,12 @@ struct nvidia_vgpu_info { u32 dbdf; }; +struct nvidia_vgpu_chid { + int chid_offset; + int num_chid; + int num_plugin_channels; +}; + struct nvidia_vgpu { struct mutex lock; atomic_t status; @@ -25,6 +31,7 @@ struct nvidia_vgpu { struct nvidia_vgpu_mgr *vgpu_mgr; struct nvidia_vgpu_mem *fbmem_heap; + struct nvidia_vgpu_chid chid; }; struct nvidia_vgpu_mgr { From patchwork Sun Sep 22 12:49:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809107 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2085.outbound.protection.outlook.com [40.107.237.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05B443307B for ; Sun, 22 Sep 2024 12:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; cv=fail; b=ePCkvNh5TR/MWX9/DRykMV9zTXeRJEWYYAiU56TjgJQv27oXDKvRLoty9soFKdmByd3zQGztwYQQuwJK80ZxYkkiFiXMAXfQxoqjmnqrIoNI7S7GDzz6d6PzLGm0ouHjleRZMw58AT21Oz7oPnQem0ktLXfkb2qlEDDdCSfgO+Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009453; c=relaxed/simple; bh=1+LjtbkZzQHYsWCSX6MmaUMB/n1hnkKLliSyUOPn7LM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KG+TXuctwd5I9V+UIqDvni79rhYC7LowEzIWeFrZdCsfprdZjT47Vi3qu4mcPxh0OmrtRBQke2EmQXNsheaXRNENidVSi/NojkdruViRCcF8JFiFvPv1tfv9LuJ03KvWSSRAmTVCTi1948ZwZNkf5tLRBERXhhcsnAI4QntvD+M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=B5ZCfmyM; arc=fail smtp.client-ip=40.107.237.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="B5ZCfmyM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=o6nFPitkvASkpmgXbalfpnmJjW0Ugpo7LlnA5QRHbNcjo//QYkXQbCUCDXJ9BiNo67qirxmoCRv4E+59RzeA3RcfjhSU6mbhKrk6UyQEVGHiGHJFoaccXvInTh2pOeoFqOPwQ3FMbNU3Z6xAdI/yc/hjJk3kZXa7KGhwIHV/RuwTnU52mJLh84tdue47zhe9LMkYzZDZzqb8QYiVYB9lnbZyQhGhFZ/FMg5X46n+mN9nTZdDP4nOYQREjRyYJqfC1HSeIVIUIS71zbd71xLEJ+CaBa2BWVpJoYGA49FIK+ybXzL4jFppEHMskN+fWpiaOcJZM7AH2615zLpYs6+nUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xlx6+VdPSIwokuwXq4IqUDgeLIIF0Xm6nfSOwGBGbMs=; b=K6bTXRE25eqACNBjHcMcvLUX0mODPJLzCf9zqWPahvQQIAu/rxzqfE9MrdPR4QoGWkfzCvhgSyZystFauaGSta40TNJhYBlMjr41VKKjJykSsJPyv34dByubJoPDqOgLzTLp+rMuTvIy0zgKAyj/rDkdxRVWAhgHkbq3Bm5hL7602fYNuXf2MVFayZZtadbHPkyhRwyOckqDfOc7nemt0M3WUbPlE5NH5n0cxZ+W7eiXnGjMdFjatWVc6/HyR7TaEh+Upzjvoa/Ga4oslWZ2ynkiZr4SHmKnM4744+sVBo1huVzGlfB8ZifuLu7F+bRt6FtfDfy44Wu/WjveFZTq1Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xlx6+VdPSIwokuwXq4IqUDgeLIIF0Xm6nfSOwGBGbMs=; b=B5ZCfmyM/aj/foX9PRKyu6TeXiUvUyNdyX+I9nt6wPdZLVLJgoUMcODwAGtZIm2sdUc6fNjhsJWK9I7L0tj1fEYOEE64Ua7r42ZFSGZ6r3W/UnTsrt5Jne+vZk9TMzCcgpGGlI2+hha2VJ5P0Uzt4Y/RQqfBRpMIqvl5a2XYTBGx8MWeG5WiRLgGQlTs3oBGjAIpe8TC4EzekziFsTQpU//fX0nv4FKrQ1nDFpcr1EGYoqxvb3PXTcwYSLXZZEw4JPbb7ZxBzVxhtiussGu6ulQaJmoVLMqlWA/abL9h916gmVYN0zFPkA3rj1MaoF5MCME9vWsG87W6gSWAiYknSw== Received: from BY1P220CA0012.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::8) by CY8PR12MB7585.namprd12.prod.outlook.com (2603:10b6:930:98::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:47 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::52) by BY1P220CA0012.outlook.office365.com (2603:10b6:a03:59d::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:47 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:35 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:35 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:34 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 24/29] vfio/vgpu_mgr: allocate mgmt heap when creating vGPUs Date: Sun, 22 Sep 2024 05:49:46 -0700 Message-ID: <20240922124951.1946072-25-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|CY8PR12MB7585:EE_ X-MS-Office365-Filtering-Correlation-Id: beeae85f-9860-4ba5-d95c-08dcdb052df6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: L2ihtOfhlD0d+VgzAZWP1Mb/MVDv07rbEU9iQO+g8FOBbt3iIhHy0NmvLuZ24MSCwW5NI9ILoyhLhJc094/Ul9c9xRPQxDJm8T5iinFjMUIfW34nILiJMeLhOXpjULqnEIz3M0pZdJ3Ut2hnxxvKzq4bDDNThdM4TF3XxRd/PW7XoQZVulm84NlwYc+9ReTQwoNbwHKWTpOB719/6LzsrwCmAkmWIfBg8KHK3s00+Wp7nZrp9ujh+WHDyEfle3tNUHq/XlVV9KLMAPVRg5NwES/443+xL61aMgO8oiLVhPuMiVnNWgxShpT+Zo93Zv+oRzVZDGmkn6g8WjPGOSo1ASLUrSa+/kU+3q0TT/vIx/vdJsIJGxZfvTRkR511hnVNcyy6UdHhFXO5f/oCjPAQMQIohGulHiXva4Z8qvRVFnWByLtS4NHQX7WRAdXu0y/h5TdJxx6H6Q6H9i4VjjVswyyL+Ojtz4Sf1p9cndotzRV1he0srbRNKq0gn0OwjAbpbKLwI2xJkeWJlbhQuIUZvk7+Q67wGP0jCFzAmDfJBIBFE7L6X8xTZyoNrOOCK9z5fC3rBukHdhMsthBE1JAgmukGnpubeEMv214TL3cyiT73nTQeE1ov8n+2cy1a4ThprHtlc89c8MqGujUqdunwwUN97PuSbS3V0VzSocNDJBSAmbi05Fn1zZmjV4zvOSpL643yZPnrLlEzv1V2sPdPSvaEvcnpfg0HhLMiBwDrbpTf+ml+efCCaQ6AC40R2LwUBxk5qMAYjrzDogbIhIo2I++QUXFLX/q16x9XCnim6LG73HITQGcZ7sCaWLisOqqzAo1oRGlKdFrZDO2YlMptAGKaaQ6qnBF1MN0CpLRJZr/f10G2DMrcxvy/5Y1tW+eTa9Z8GHzaP2XNsGdko3ytWcR1pXWqyi2WHXS5DcWxnK2d4K9QhOipf8av+j/yGt2nF/+jN3Skg8ga8Caqxp9iDZ4I9T4+spCUf05M+mEDYdBipJfAg8mXIQK0dtgHnbnyYrpFlErtcMGeij6NJFM9rFePyHx4ZnhNF9FpV9QfYhIIpar36ROAA0Kautd8z/dmJM+NPR7igdClLhDMvOTg8F/iwjSYIU7onacmhqMQEukSrvq8x9m7X0qSEIskFYh+bCWDgJac3tOshEIWM9Z9zJf6F/Gf62JgRpsBLvPE2Cx9QRBv1wULpFknP2gHrZ1yrDrnrzh7FbvADHiyPLyqbBNdzt+j9mqup96SyAAheDXwi/CSWzna0uZsW8Me183eJGwkhUiRXPd2swq6wJe7UyTJN2YEovAuni0F8nOda4J3Ms3ZFXw7d8Fg8/zOKZJz05wZU+nDz/sMInPwVk/FrZUnfzcDonoUiYFFyo6eQZ+6TuJ9qW3Lt8HjmyIIDbO4MhC8yzVQ5KJurYiGo2DN3h/9x7V10BZtvz2xhORIuVmllDvImweyLdq8LGszydqs X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:47.1458 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: beeae85f-9860-4ba5-d95c-08dcdb052df6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7585 The mgmt heap is a block of shared FBMEM between the GSP firmware and the vGPU host. It is used for supporting vGPU RPCs, vGPU logging. Creating a vGPU requires allocating a mgmt heap from the FBMEM. The size of the mgmt heap that a vGPU requires is from the vGPU type. Acquire the size of mgmt heap from the vGPU type. Allocate the mgmt heap from nvkm when creating a vGPU. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 6 +++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 32 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 6 +++++ 3 files changed, 44 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index b95b48edeb03..50b860e7967d 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -76,4 +76,10 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_free_chids(m, o, s) \ m->handle.ops->free_chids(m->handle.pf_drvdata, o, s) +#define nvidia_vgpu_mgr_alloc_fbmem(m, s) \ + m->handle.ops->alloc_fbmem(m->handle.pf_drvdata, s, false) + +#define nvidia_vgpu_mgr_free_fbmem(m, h) \ + m->handle.ops->free_fbmem(h) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index c48c22d8fbb4..4b04b13944d5 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -87,6 +87,31 @@ static int setup_chids(struct nvidia_vgpu *vgpu) return 0; } +static void clean_mgmt_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + struct nvidia_vgpu_mgmt *mgmt = &vgpu->mgmt; + + nvidia_vgpu_mgr_free_fbmem(vgpu_mgr, mgmt->heap_mem); + mgmt->heap_mem = NULL; +} + +static int setup_mgmt_heap(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + struct nvidia_vgpu_mgmt *mgmt = &vgpu->mgmt; + NVA081_CTRL_VGPU_INFO *info = + (NVA081_CTRL_VGPU_INFO *)vgpu->vgpu_type; + struct nvidia_vgpu_mem *mem; + + mem = nvidia_vgpu_mgr_alloc_fbmem(vgpu_mgr, info->gspHeapSize); + if (IS_ERR(mem)) + return PTR_ERR(mem); + + mgmt->heap_mem = mem; + return 0; +} + /** * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance * @vgpu: the vGPU instance going to be destroyed. @@ -98,6 +123,7 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + clean_mgmt_heap(vgpu); clean_chids(vgpu); clean_fbmem_heap(vgpu); unregister_vgpu(vgpu); @@ -139,10 +165,16 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) goto err_setup_chids; + ret = setup_mgmt_heap(vgpu); + if (ret) + goto err_setup_mgmt_heap; + atomic_set(&vgpu->status, 1); return 0; +err_setup_mgmt_heap: + clean_chids(vgpu); err_setup_chids: clean_fbmem_heap(vgpu); err_setup_fbmem_heap: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index 0918823fdde7..f4ebeadb2b86 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -21,6 +21,11 @@ struct nvidia_vgpu_chid { int num_plugin_channels; }; +struct nvidia_vgpu_mgmt { + struct nvidia_vgpu_mem *heap_mem; + /* more to come */ +}; + struct nvidia_vgpu { struct mutex lock; atomic_t status; @@ -32,6 +37,7 @@ struct nvidia_vgpu { struct nvidia_vgpu_mem *fbmem_heap; struct nvidia_vgpu_chid chid; + struct nvidia_vgpu_mgmt mgmt; }; struct nvidia_vgpu_mgr { From patchwork Sun Sep 22 12:49:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809110 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2047.outbound.protection.outlook.com [40.107.101.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F274A3CF58 for ; Sun, 22 Sep 2024 12:50:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.101.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009455; cv=fail; b=QPI6ShV+pvb1+uWQNobBeaRhmKOgD0VTVIlIqTMlYIhWb1UwmQjPhUczMh3+1czYO5K8ZkMzgcq1yNy0bYlk79HouEmk2vkeyrkshGYkpDwcy3uMscYO8I2UtVEAPmOmxQKVi+2kE9Z1zH98cu31Y95vdtcIGbiSNS+mYcdt694= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009455; c=relaxed/simple; bh=nnLQsphBrO+nZHTUEkfHTBTiEBt5+mV//4sEozTqfYA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=riLeZcnmjr+OOgt33jZxKpejpDQ5f+Q8rb7x5THL28YqnDqUKvBW26qTxyOpiiesV1Tq7/RFRBXWH4OGtp4lI48jPp2vaDBqvMMyhZd1x1l7wGl7eESlp2UjLUCvYRaXlbED1KwJzIopBXtA5EUl3lnkSxFXANU6OTE155LUhos= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=kHDgVaN9; arc=fail smtp.client-ip=40.107.101.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="kHDgVaN9" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=W6zJ7lN3pl6fcZQKuCvNsPpElJ9yTGh7uT/+LPhyXi+iHW+Hm9d+ND6jFBMn41TUHL1Ht6sEtyEJ7IeZAX6OHPDClvgJmS65LkTWZxI34qhNn/eo53IkuhwhuEl4nwg7m40HjpmKnab63wTgO6RIv+RTtqAlTzXfPi4Bkx4LDmU+GfwnsIQEhWipamo+JQrn1PFUG3D4VkJ7WEGZuUiDhjN1IIi5ZVxnmc2F+D7bowj+pMNnvZHh0L5WfU254URPTbBx7ftKlDmtdOcbN1ZbqRZS1HufcPVkTkpcmPel+L94Mv2bQrhYt3+4fGJXvrz2eIWbRB7+Hg/VMZGWiAIZUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pwBqXj/y0il+KibBUa14o5qJf0LgGdlyPP0GVh9Vf7I=; b=paYm6k3bD6+orh1b+vIKSCbMxLMzIDAaXOjDCoAscz2vJXU7KZeOCthx7Dspr1DmnsaxiJ8Yn3ds23jOlBPf+TWeF1qMCE5pKKtMKXZUR326mwL8ospsDlrSZIblj/o1pLQL0+KmoClyKQM/MQ2WAotXyzOg1WX5om9IjF5CkaMwenGmLHhGElLIO7kNNftJX50ypUXzqoUaUUdgZ9iPvLmDBWgVxhvcUnlOQhgMzEg6YqbCWJshjGLSVN9hyQg/DspJDfaeMP2e6Rm+a5Ntd6r0kYtsyOfTnI1heJ5VUsQoVAL25RCaOMtAtKRGewSJliaV0gud/TMXGrnMO/tDGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pwBqXj/y0il+KibBUa14o5qJf0LgGdlyPP0GVh9Vf7I=; b=kHDgVaN9zRN5DxFwyqfG3ppzPzSrwtGw64bKALeh42yw67ixDOhhrWA9HEx/t6M86199ZT8lF8sBKJ/LEQ+mQQKKGB4nM3r5sZPY3UsG2A7W5iuAY+2L+90pSxIoz8rMLQbv4AswaCxNUP8plu3nNZN3jpVNsyEDnXSPpxX9lmqJfwuLOmGdAdUk2yE7O7KMMmAlzV1aRYKK2wL2G5SMBWkABdb8ZtxOu4qDQGBtuvV9SuhLtExUS6D2avZZGUHDG9FwFXF9UHbFa/U6vxtdI5jhIu9C40v3YuYTYgemKQ+w28yaE2Txvd6iuRXHGeXSi9citslXCCVkSkISQe0trQ== Received: from BY1P220CA0008.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::16) by PH8PR12MB7254.namprd12.prod.outlook.com (2603:10b6:510:225::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:49 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::7d) by BY1P220CA0008.outlook.office365.com (2603:10b6:a03:59d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:48 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:36 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:35 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:35 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 25/29] vfio/vgpu_mgr: map mgmt heap when creating a vGPU Date: Sun, 22 Sep 2024 05:49:47 -0700 Message-ID: <20240922124951.1946072-26-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|PH8PR12MB7254:EE_ X-MS-Office365-Filtering-Correlation-Id: 3de9d63a-89bb-4f69-2f0b-08dcdb052f10 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: SqD3alNH0UUKDMU+Ot/W+8MNpvkek1BRKG4HzsHH6mS5e4xaFXRg3IfzV9U2ropQo49EHXvCwgVGVL2CEk9KP9wPxIxRPu7eo3F39B4LhxhDv8G1CC+w8aKf2pBYOLtfMop9fwKBKALeDDHNuFqCWzNigQh9pH+cst+pwEsfQvqh4z5BBLoObTuiAL29p5YQH69+0m4+FXZESM+odnueMrcmvwPmEuUh+LEYmxZigD4dJpihDSZPXxl0Ioe+gLKNZ1JWGlH34LlUfUDQFSillRS66N1DUjqk7ml50ogapMkNnDPOElyZE/Nw++14NS1KtZ8Gt0Ffpc1KEq1XVL9XltOlbhggiy+u8/I66XNcdkLrzchshMZdSo5ltAQwj4IfpdsgYtEs+nSWY9eOCeiRii3K6AhHZa/U3P2ZknTUqb5KQylx2u99cP5arsu4r6pduDJQUM32Q/zBGTreWSB1xyvASAM59y6M2DCtrTo4m6to8zb0TZLad7leU5VY9prgF578q2tzNp6GDUWmqginzYcVCk76Ysc83fY74XKZcf/gQhCY/v2XAAgHlQG3O5FWRVeVuwq902j8d+PwnhsWQSS6XPjEpzapWFw/38eLvdeeUD1zOh/4jN2771JE/hxrUj9sm4+4OoPO6sOe5taOnQxmboO1z39sAetElsvhGO3GtVckNXYTzNyVtY8wYRFWK84Yt7UPQE6VPQlQR6J6kaQEMOdDZZcvqfd9DYY5s5A/8Ae4/sUr6LdZ+aCBayoxNnxMcMunElbdINadgeXf+xk+NkdbIntuuUVLv/QWJnlY7CVXaaIlNVuh18xFgmoCT1H5pjSEcpw//JP0pHxfFqdsRiA9KRcxaNyRscK3UULHGfZhRN9ImMJi9OJGChfoSlcNdK91wqeTQpUQgl5iBP/Lkis3rJ36K7an8fv/NiQnItYsnu3PCTx70K6Zsnj85ZT6j1H0NnQh18JTqEVYhVERVoOhlEywqADYa+KwSlRCbWpwK7gQrmPzaGLEjORRVH06pp68j7UCQ6zgbFHCiyKYou/X0PtlOc6/Fvh29N29g4cNGtNSG5Kz71W5OLJlgxTdzmFq/paMBMWCioXdehkpwmzkSNI6jCmkvldCP3lb0n/BQhuu2x+bow3fErEZJ2k2ig9l97Bj9ZhRjDfrhc7f2Xk6JqjQy2y6C2McjgL34YhCxEJwlmXQ1tBL6TJ/6xOxbJgN2Wkd5XlQSFo4Iu9C2aXwUE9hSWpLkBWrkIjultrVqE/5Dgyivjfcps6eq2x6aNHMOyKtCrfMJ1zZ1R4+SrNsY1Lpk34xtYYNS54m6j3Mxj5CxfvW/8IFZmP658mWiauYTw/MneSCCNErOcye5p/DqDvaqjWkgBCD4DA= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:48.9895 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3de9d63a-89bb-4f69-2f0b-08dcdb052f10 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7254 The mgmt heap is a block of shared FB memory between the GSP firmware and the vGPU host. It is used for supporting vGPU RPCs, vGPU logging. To access the data structures of vGPU RPCs and vGPU logging, the mgmt heap FB memory needs to mapped into BAR1 and the region in the BAR1 is required to be mapped into CPU vaddr. Map the mgmt heap FB memory into BAR1 and map the related BAR1 region into CPU vaddr. Initialize the pointers to the mgmt heap FB memory. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/nvkm.h | 6 +++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 29 +++++++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 4 +++- 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 50b860e7967d..8ad2241f7c5e 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -82,4 +82,10 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_free_fbmem(m, h) \ m->handle.ops->free_fbmem(h) +#define nvidia_vgpu_mgr_bar1_map_mem(m, mem) \ + m->handle.ops->bar1_map_mem(mem) + +#define nvidia_vgpu_mgr_bar1_unmap_mem(m, mem) \ + m->handle.ops->bar1_unmap_mem(mem) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 4b04b13944d5..de7857fe8af2 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -87,12 +87,29 @@ static int setup_chids(struct nvidia_vgpu *vgpu) return 0; } +static inline u64 init_task_log_buff_offset(void) +{ + return (3 * SZ_4K) + SZ_2M + SZ_4K; +} + +static inline u64 init_task_log_buff_size(void) +{ + return SZ_128K; +} + +static inline u64 vgpu_task_log_buff_size(void) +{ + return SZ_128K; +} + static void clean_mgmt_heap(struct nvidia_vgpu *vgpu) { struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; struct nvidia_vgpu_mgmt *mgmt = &vgpu->mgmt; + nvidia_vgpu_mgr_bar1_unmap_mem(vgpu_mgr, mgmt->heap_mem); nvidia_vgpu_mgr_free_fbmem(vgpu_mgr, mgmt->heap_mem); + mgmt->init_task_log_vaddr = mgmt->vgpu_task_log_vaddr = NULL; mgmt->heap_mem = NULL; } @@ -103,11 +120,23 @@ static int setup_mgmt_heap(struct nvidia_vgpu *vgpu) NVA081_CTRL_VGPU_INFO *info = (NVA081_CTRL_VGPU_INFO *)vgpu->vgpu_type; struct nvidia_vgpu_mem *mem; + int ret; mem = nvidia_vgpu_mgr_alloc_fbmem(vgpu_mgr, info->gspHeapSize); if (IS_ERR(mem)) return PTR_ERR(mem); + ret = nvidia_vgpu_mgr_bar1_map_mem(vgpu_mgr, mem); + if (ret) { + nvidia_vgpu_mgr_free_fbmem(vgpu_mgr, mem); + return ret; + } + + mgmt->ctrl_vaddr = mem->bar1_vaddr; + mgmt->init_task_log_vaddr = mgmt->ctrl_vaddr + + init_task_log_buff_offset(); + mgmt->vgpu_task_log_vaddr = mgmt->init_task_log_vaddr + + init_task_log_buff_size(); mgmt->heap_mem = mem; return 0; } diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index f4ebeadb2b86..404fc67a0c0a 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -23,7 +23,9 @@ struct nvidia_vgpu_chid { struct nvidia_vgpu_mgmt { struct nvidia_vgpu_mem *heap_mem; - /* more to come */ + void __iomem *ctrl_vaddr; + void __iomem *init_task_log_vaddr; + void __iomem *vgpu_task_log_vaddr; }; struct nvidia_vgpu { From patchwork Sun Sep 22 12:49:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809114 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2067.outbound.protection.outlook.com [40.107.212.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50A724C627 for ; Sun, 22 Sep 2024 12:50:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009461; cv=fail; b=rCyRm7aWqWPdqmnvagZy/gyHQ+HghyQNMC6/L0sI+KVVKu7622AJWc77FQxoDaPd3cOvZ8rxu0JylDJ5gozof9UQvCwrMGlFZ5dYyJuhAhoqvZYYYUNsCMBhfQFXsS9qyMpPRfQ9eRTH5ymt7d+4jmgwYpb9w8rFKizQb+7QuFw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009461; c=relaxed/simple; bh=NujKwmUego+xcu3y+yRH1s36MrLeJMCUOAp9gCyg7xc=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BulwM0via2HPaL3gGe6xejBfVHu4mjwCCgtgQk6VBJZWHsTd83EpSIcEu8uWLyBXJswt03VLoIA5NPubgoTwvXJd1UcP/hdP6IWlCWI4Ro5kqPwXD48Y2eENFA530JdzJ8lT8wlx3kBOEPmazoFm7ONiCT84cNFYwbZONt/LKDk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=nkJ+Sr1L; arc=fail smtp.client-ip=40.107.212.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="nkJ+Sr1L" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=c3NRnIJVVi6fZbngUZNx7uoBQY//fpG8X7TlmLL59O+ylxihUmulJiBjOR19VLKG7CjvSySNHr/GKTlTcqXaj6lA4jBWQiR5TEXqe4rF+jZCNekc7rQ9AukoeX6wveAIW+Wt+p22x1F6Gi0l7d08RhnIJsxBr2HUzlDqbbbWZ2z+B5j9GV7wM5IgUF1VMprCz9JmFzweXn4E6550Dp7QVqsfRDQ7kFcy95Sa30dIe4r3QzndZct2XsfmD+pefW7ZjEJGBiARu7vcBZ5vyFlYzFTmsL+8q8nvxgMIjY+yxzOG0fP8VjVJDFuwuRQaFHM4RfCdZfIMJ+eaDNmqK9OZ5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4zFR+jTfKH04tmYtUSihJrOSgRvqGnHfZmRY6KjZvHs=; b=cC0YVFkyjdAOaWeuHQ3p+b8Q7kxO9grvYzpYSqnIklijCz++Eb48+BGnnUuWt4RvB1833GLSQDH57l9ig3DPuRVrNoGpVSJqqvA7fTNzuhaeMjlSM3BOj/7I5a6VQQdW9Ol5/Zid9BNgC6LVQQLPdufbfWOyMhndJ4srBC7DTRAz54GYrbhfnKHOzY9VtzJk1v9z5MjWmV3lRZWjztl+UCJFp3+UvwBlwG6N068p/TTfiWPyiq9ndgdLy5VMBLbcCSjB3SULAISnN5PJYqbrcpnXiyp939mq/whkCVRx+BgFvo1hXpZz5Ah84q25qzXfp+J280jcmSEXWeuFM0k9kw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4zFR+jTfKH04tmYtUSihJrOSgRvqGnHfZmRY6KjZvHs=; b=nkJ+Sr1Lem+yCUI35KG20hSdXkbqAegxIkWFzQAVyOzFBEVEw4Kkdoqtw1kkpytFO/qoX+acErmizfcNCsl6IcTm5P6m7eIsKAdwfcFssGpPPp/EsBKR29+Q3AP0blfJc5+w50V73f73QwLzW7g5AZzlQVhq9KvWT5zL+eFt+kq3cZIrh29XHcZkEuRarfT7yXnsyYKGEDjQbxUeC782mM6Cx31+leD0sGpzGe9l22Eye827+R7dTgJrfjCnXZQovsEdTdTiin+dguNDvzJSna2IQIvpE6vB14KvX/v8BaDGfv8x7GHdii043zWZl6fGbeQCS+uwcoplj+mmXKoDBg== Received: from BY1P220CA0012.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::8) by IA0PR12MB8975.namprd12.prod.outlook.com (2603:10b6:208:48f::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:52 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::c1) by BY1P220CA0012.outlook.office365.com (2603:10b6:a03:59d::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:51 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:37 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:36 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:36 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 26/29] vfio/vgpu_mgr: allocate GSP RM client when creating vGPUs Date: Sun, 22 Sep 2024 05:49:48 -0700 Message-ID: <20240922124951.1946072-27-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|IA0PR12MB8975:EE_ X-MS-Office365-Filtering-Correlation-Id: 2a803d22-b54c-43e9-7f01-08dcdb0530a5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|376014; X-Microsoft-Antispam-Message-Info: DbNXr8eqeciymHsUMlQYBKVt5253IOhKBCEWcdHR7BreYzd1WA1sQmmHMs8/XQH8KaKVmtyv55XM+pddsvBjHboFtjLsKdjJrGIQQa1kaWRooDT2LbkWs2y9eZQb3wO5J4+4FQF7OXpvRygLTYH+5o0GiXR3rshvUVm1JZDpCyNhaOnA9m+ew3dU22Hm315KAW+URdiO9Ix3uc/fvqGscBdBi9cEIJ0uZ9wUBwF2i3ITceEcSnnqgY+SwmeuafxmRW0pa3KoR924K2qQH8qtiC46H7NSm/GIAGrP5wxVq129v9WsSrFncv6uA9HORgpzD/Tq7mhCYoGuQo4KqCMFwpn0JTtntr7lNTaAczT0Q+8rZe9IF8tx3mUsWro4qIAOn2wb1YAaDHgLMW3QkzWXl7vEb7Zss3yE0LMQDheBUii6GEogZxcFDnfzSk2xEl0MtbOXMV64J5D+45+IAsc4UceyRcwQWyLfQIYfz5ssZALXwqVoG9TpBF9bVrkyJ0hgGCMSekfZaOgW6+Dqqs/yrIWglPR8xXHge1l8B5pjVh/zURygftiGUOzraTroJgHPB0nVbeWzfCVR5hRTJop/W79FlRErI/9gbgLtWmvYlOt1iukeyUwulwbftK6GBIF1hFBa7eWPlWAV34d6eLs+OOJFAmZeIVIY1DibC9HaJQxmWZTe8YEWcsTgkCGL7PajGgijCclZ75qdfyGUhubveE9HEMoiwstJP3SOXV+XI5PcVGPJHjgeVOlX3zVIDW38INo1OO+Rq1NJfRvHNPoxXAPnCTMm09eDuNNMk+gJC58Ugh42XiljOcNHrHvJ6eEHS9NemSXC4ZKtVlW+wwXeVqULL+JOZdYNRrRMRQi69EsTqt7tilJNLZ6rKdxw1DfYsvWRBTuRioomEqb7r4rkx8LumfKxW010uSW4lG/R9nyTzGqtdNCIgscHEbTi+BZXMSK/oVp+tdvPMqIMMhc+zYK99J1mC3CDhetDZ8aa2hMc/EHVtOO7FDW6jY/kVqXNSRXzxAqqHqYJVm97FStK7Vt/zrJHQoWbz9WoDAfb85GnJ74d6ukWzI7Ou0LfLZHgQIyBiVhAYuJ/kAmh0vNua4wabAuz2m/nlFnY6GoqMfyVla13lOvjeLsX4LgE1Ez6bY77aPyg9Kwo3M/DEyCJCswqGcb1rynrT/TMQ+BtfYlNhWn2RDlbzEQBtzkyoXsJXmsVCDn8NXTopjutHQ82yevNTNmj8QdK6K6r8c36EkR4CZqkWoDg28wOZFWRjqb88KJ2hHFLqSUHDxCKKPlqcIuYQa82eKwYU0JvhKeSrpsbhoS6n4XAjj/6U8pyRm94pkXG3M9zaExt0bLHeCWJ6Qdf756PsIhfXnmVFm3N6P/6BS+h7kQva9Ig3TqlMXgX X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:51.6458 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2a803d22-b54c-43e9-7f01-08dcdb0530a5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8975 A GSP RM client is required when talking to the GSP firmware via GSP RM controls. So far, all the vGPU GSP RPCs are sent via the GSP RM client allocated for vGPU manager and some vGPU GSP RPCs needs a per-vGPU GSP RM client. Allocate a dedicated GSP RM client for each vGPU. Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/vgpu.c | 11 +++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 1 + 2 files changed, 12 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index de7857fe8af2..124a1a4593ae 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -149,9 +149,12 @@ static int setup_mgmt_heap(struct nvidia_vgpu *vgpu) */ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) { + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu->gsp_client); clean_mgmt_heap(vgpu); clean_chids(vgpu); clean_fbmem_heap(vgpu); @@ -171,6 +174,7 @@ EXPORT_SYMBOL(nvidia_vgpu_mgr_destroy_vgpu); */ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) { + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; int ret; if (WARN_ON(vgpu->info.id >= NVIDIA_MAX_VGPUS)) @@ -198,10 +202,17 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) goto err_setup_mgmt_heap; + ret = nvidia_vgpu_mgr_alloc_gsp_client(vgpu_mgr, + &vgpu->gsp_client); + if (ret) + goto err_alloc_gsp_client; + atomic_set(&vgpu->status, 1); return 0; +err_alloc_gsp_client: + clean_mgmt_heap(vgpu); err_setup_mgmt_heap: clean_chids(vgpu); err_setup_chids: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index 404fc67a0c0a..6f05b285484c 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -36,6 +36,7 @@ struct nvidia_vgpu { u8 *vgpu_type; struct nvidia_vgpu_info info; struct nvidia_vgpu_mgr *vgpu_mgr; + struct nvidia_vgpu_gsp_client gsp_client; struct nvidia_vgpu_mem *fbmem_heap; struct nvidia_vgpu_chid chid; From patchwork Sun Sep 22 12:49:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809115 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2068.outbound.protection.outlook.com [40.107.95.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E76285579F for ; Sun, 22 Sep 2024 12:51:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.68 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009463; cv=fail; b=FfdfD597+tJS5AlTOd5ZE3ZHrH4qsAvgid7qOIW3RiyMCPDKGtbN2V9Mr3C14v0qMuvtuuxgIlvbVPetE/2vgegjZhebcoTusJSgEDxiePgtfWyXDFsAi004e2yfEwi5kNFXT2+l+WDkcSpLoEb+rhxFp5c4BRuolysXOR6n8lg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009463; c=relaxed/simple; bh=DklQhxhCW5x6FdxHtGiK0OVUal8h9o4jpYNMr4GXY+0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=coaD6mcCJGAdzzVYypJN8qOKu+KRiI9BvufE2kt6X+trW8eG152xF2j+Zq8F57O/nkLG/7fuUsUYuOUXD7jOETT4Snz09gXCVE7WjsCN1oS7EreecBtGAQqbw/+whWHkxT/nJI6FcGgxdzeUv3tUxWRF5U9RXoKk3dtJYAGAfME= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=clmClB04; arc=fail smtp.client-ip=40.107.95.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="clmClB04" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fgpXrMsGBLDxKc6HxbfmCKfYFgfKJN4VZXRfvikzPNBzDTDKKXOLk2sPI7tEK4QrnE4v5kcYNYdLTPHovzNJw4jQ2WcBb2PxHdTk9e1awC+HCUgK+AsVuEVjtvSkW4vwwnSxgk6/S6yZsSGBtELtXQ4kL4AWh2KF0PSZ8DbV5zb8K7w4Et9IOTyMHiJeLabypgandEQk6VvrVrxWBiW0Hp9GfpoWwJoplimKy/OJCUna7JTxh1AQxT/Xpe1PRClG5A167AOyCiov4lCJ5ZSCcINoJEqETFNqs997kJXei58NTkaLT9uAC9VWodAD8jzJl4sDOs2Z3rq4xGof4aqdwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ra+LDNCHYJOxJt2WQhpgZE/Fu8ztPBgzR6AGaxUZU8Q=; b=Q3/ytZH/nMm8Ha7lyOcUDIVnuFTwrOUbl5yx7fleIRit3+1mNtNQL7HFLFClwIemqjoPOQvFWwiDkhartq5tgZotrNyChZYjVX7Q6IYK9TSi+boWXVCD2Zsviti0ruYqxRl6wwSHG1H+lVpulupJ8Kx1l3fRYSJacnCu6VSo6+ipdd79zvSmOvnNbPP5v0kM+ZYn5pK6/IUHbH8TKSRFLOvMs+VQBr1+9lX+iroYHS2I7szDrccKNH4gGEc9c8WAk55AUCc6HytKAYxrwktp3wZ3qKX1vZiucuVoXEqkJN9Ng23/4/8YjOTM7Acf4qEoceanhJsIagDTgSK1XD3OrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ra+LDNCHYJOxJt2WQhpgZE/Fu8ztPBgzR6AGaxUZU8Q=; b=clmClB04nEmMWq09MOWmIB7lM8tT0wUU1VhBJ+HuUGBoktWg097hw612n60Vd2cwHLF4/InK1hzYo7pfKTc3/AxSvy2/HpwkExYACRy31lsfGOmal6WRGZ7bfW2DFMjnX27ieqgDxFmhQnC7VvEYCupq/MI7v1BuoWadcu84SchyZeVF5Ni+rUJgk3dU5jvjRbEL3r0WKDCh3Pn0/P2ChOyqJKEZhPgr7djh+upeQP/q4KRJ8J2b7oKeaWM4TpgUtn3ndYquOa7lPqTbgBMWvsYwRiSqnsDWDzJukNod9z3rLKKHbwc0JRsWr6yt3U+r5tjSvM3gTGCXV3b8FWgG2Q== Received: from BY1P220CA0008.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::16) by SN7PR12MB7369.namprd12.prod.outlook.com (2603:10b6:806:298::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:50:55 +0000 Received: from CO1PEPF000075F2.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::38) by BY1P220CA0008.outlook.office365.com (2603:10b6:a03:59d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.23 via Frontend Transport; Sun, 22 Sep 2024 12:50:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F2.mail.protection.outlook.com (10.167.249.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:53 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:37 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:37 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:36 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 27/29] vfio/vgpu_mgr: bootload the new vGPU Date: Sun, 22 Sep 2024 05:49:49 -0700 Message-ID: <20240922124951.1946072-28-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F2:EE_|SN7PR12MB7369:EE_ X-MS-Office365-Filtering-Correlation-Id: b9dd9356-c6f3-45e6-fa32-08dcdb0531e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: RVmsHHERDBd6bpjPnCXASapwxqEMOkfMNJYk1LN4S3AehMh8w64b5/pz0SavWvEZlwBmGF9ktz1uL89pwtWdxvtYOqntrL7yau3nkwWooVbmDu0MkQKX6ZikwdD9AJo6AghDNsQb93/0rH7PgnmQGoBNT4HaHeq7BFasAohoBreMdqhPHJ1jvI83TWMMA9eLc5EDGmEZ6Wi80gOApk7zXsS+Y9F4Xk+HQLAuqCSwmDkl44JcdB3/LiAot16J6p0YbKY8YDF9s9G5n1tP10YdgJ9LhcqDh12Vkeo1lsd039D2Ec0TukdCPqdBlUzfeOlD+ofamY8iJuzmAm0RUAJXy4bh7cT+JhfgCXjF0KHVU9Q8YSZOP/Aw4DXZnOtKb5fp9fnjVe5h7Vf0EFqU4tTaA6oyf+Xqegf8vIus6YrZwFronjaP+jZatF235rjAHpXbJ6tWGdV+rwwBSLKOxUuoWiEMUZFfb2IYQskGVwTpgRIfHQlxJwf5xaaag87lexR0e7M5LuHNZJTAZHtes5244g97en78lWkqKM4DQUfVwt4TirKPoy7WxteoFv0n4RZcffbJPBCqWzUkolPad+g5uUcTb7Fe9Etj0ZaRnUsNMYdm6hqMaE77E6G+Gh2OQLdcQzrN/GDRI/4ck7N/Oer0UFMC824uwY7dGoGuDue73TQFg3hCH6sghMDoIhSZOjDvdv1kfWzA58HAMoarUn+x0a8/Cy6A3W7V1aft0+aztp4TG3dHpFedPfVyWpNXllE7RxVhT6z3H+GZaOXK7DW4uQNs2ZRtF3A+9Ey4pNagdICJsY0nqz24Ver9JNTDSZMQA/4Mtx6onqLA6xWqU9P4Esy4lkwpzGsl+3/qQvB2JH8Lpfm36Ri8I5MCwJs6JEaoy18abo9JTdLlGgJ1ByndCFbC/KdyQ+/XouonaKCqXnBmraKb7AoqtHy5nBFOGmpA3FFGhHpBV0AePYjRaH7yFIoIQy52IjLZTeu3+cN38hxKl5PYBgcxTNn4xTgcSIeFU1umNasgofj3AzGhUCXcYovnQb3PhfkdZSnhSYB2pVmdYGp555r7/5YRYBPZwhdQF9e5rIuWBKA7/XyHHJTHCNBOk+Ku7T9mwdzZ3Fk8utCaXHNXh+yqTH1ozkyp+xrg2Q7r3LW92eDVH3uSdvuHb+0m+Wct8Y4HA0suBW1jDrVqYuCdMPZ9h2s1lpjr8aSktREVK/CuOkwBvVPrXb/eKhMqoNF+gQFDm2WnMm4R4kfyRjwV8CBSOTBHqztTcSkXfV7upJGTwYd+w447Gna/FXPRcIsK6d56T6w/L+cJ/dfPLuCjNn4O+lylhmjC0rF5qmX6jwL17AVbuXLthf52HbDnAdIyJi1g1Myvj37dOol446nPBDvSc7gvUUUAiKyK5e+30kLKWx+lIaynJSAZO6wEMHuSk/3Op7UxVdOFgsubKIziu7o4M2MUP2Rx3C3R X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:53.7395 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b9dd9356-c6f3-45e6-fa32-08dcdb0531e4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F2.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7369 All the resources that required by a new vGPU has been set up. It is time to activate it. Send the NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK GSP RPC to activate the new vGPU. Signed-off-by: Zhi Wang --- .../ctrl/ctrl2080/ctrl2080vgpumgrinternal.h | 90 ++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/nvkm.h | 3 + drivers/vfio/pci/nvidia-vgpu/vgpu.c | 94 +++++++++++++++++++ 3 files changed, 187 insertions(+) diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h index f44cdc733229..58c6bff72f44 100644 --- a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl2080/ctrl2080vgpumgrinternal.h @@ -59,4 +59,94 @@ typedef struct NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS { NV_DECLARE_ALIGNED(NVA081_CTRL_VGPU_INFO vgpuInfo[NVA081_MAX_VGPU_TYPES_PER_PGPU], 8); } NV2080_CTRL_VGPU_MGR_INTERNAL_PGPU_ADD_VGPU_TYPE_PARAMS; +/* + * NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK + * + * This command is used to bootload GSP VGPU plugin task. + * Can be called only with SR-IOV and with VGPU_GSP_PLUGIN_OFFLOAD feature. + * + * dbdf - domain (31:16), bus (15:8), device (7:3), function (2:0) + * gfid - Gfid + * vgpuType - The Type ID for VGPU profile + * vmPid - Plugin process ID of vGPU guest instance + * swizzId - SwizzId + * numChannels - Number of channels + * numPluginChannels - Number of plugin channels + * bDisableSmcPartitionRestore - If set to true, SMC default execution partition + * save/restore will not be done in host-RM + * guestFbPhysAddrList - list of VMMU segment aligned physical address of guest FB memory + * guestFbLengthList - list of guest FB memory length in bytes + * pluginHeapMemoryPhysAddr - plugin heap memory offset + * pluginHeapMemoryLength - plugin heap memory length in bytes + * bDeviceProfilingEnabled - If set to true, profiling is allowed + */ +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK (0x20804001) /* finn: Evaluated from "(FINN_NV20_SUBDEVICE_0_VGPU_MGR_INTERNAL_INTERFACE_ID << 8) | NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS_MESSAGE_ID" */ + +#define NV2080_CTRL_MAX_VMMU_SEGMENTS 384 + +/* Must match NV2080_ENGINE_TYPE_LAST from cl2080.h */ +#define NV2080_GPU_MAX_ENGINES 0x3e + +#define NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS_MESSAGE_ID (0x1U) + +typedef struct NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS { + NvU32 dbdf; + NvU32 gfid; + NvU32 vgpuType; + NvU32 vmPid; + NvU32 swizzId; + NvU32 numChannels; + NvU32 numPluginChannels; + NvU32 chidOffset[NV2080_GPU_MAX_ENGINES]; + NvBool bDisableDefaultSmcExecPartRestore; + NvU32 numGuestFbSegments; + NV_DECLARE_ALIGNED(NvU64 guestFbPhysAddrList[NV2080_CTRL_MAX_VMMU_SEGMENTS], 8); + NV_DECLARE_ALIGNED(NvU64 guestFbLengthList[NV2080_CTRL_MAX_VMMU_SEGMENTS], 8); + NV_DECLARE_ALIGNED(NvU64 pluginHeapMemoryPhysAddr, 8); + NV_DECLARE_ALIGNED(NvU64 pluginHeapMemoryLength, 8); + NV_DECLARE_ALIGNED(NvU64 ctrlBuffOffset, 8); + NV_DECLARE_ALIGNED(NvU64 initTaskLogBuffOffset, 8); + NV_DECLARE_ALIGNED(NvU64 initTaskLogBuffSize, 8); + NV_DECLARE_ALIGNED(NvU64 vgpuTaskLogBuffOffset, 8); + NV_DECLARE_ALIGNED(NvU64 vgpuTaskLogBuffSize, 8); + NvBool bDeviceProfilingEnabled; +} NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS; + +/* + * NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK + * + * This command is used to shutdown GSP VGPU plugin task. + * Can be called only with SR-IOV and with VGPU_GSP_PLUGIN_OFFLOAD feature. + * + * gfid - Gfid + */ +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK (0x20804002) /* finn: Evaluated from "(FINN_NV20_SUBDEVICE_0_VGPU_MGR_INTERNAL_INTERFACE_ID << 8) | NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS_MESSAGE_ID" */ + +#define NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS_MESSAGE_ID (0x2U) + +typedef struct NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS { + NvU32 gfid; +} NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS; + +/* + * NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP + * + * This command is used to cleanup all the GSP VGPU plugin task allocated resources after its shutdown. + * Can be called only with SR-IOV and with VGPU_GSP_PLUGIN_OFFLOAD feature. + * + * gfid [IN] + * This parameter specifies the gfid of vGPU assigned to VM. + * + * Possible status values returned are: + * NV_OK + * NV_ERR_NOT_SUPPORTED + */ +#define NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP (0x20804008) /* finn: Evaluated from "(FINN_NV20_SUBDEVICE_0_VGPU_MGR_INTERNAL_INTERFACE_ID << 8) | NV2080_CTRL_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS_MESSAGE_ID" */ + +#define NV2080_CTRL_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS_MESSAGE_ID (0x8U) + +typedef struct NV2080_CTRL_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS { + NvU32 gfid; +} NV2080_CTRL_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS; + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/nvkm.h b/drivers/vfio/pci/nvidia-vgpu/nvkm.h index 8ad2241f7c5e..8e07422f99e5 100644 --- a/drivers/vfio/pci/nvidia-vgpu/nvkm.h +++ b/drivers/vfio/pci/nvidia-vgpu/nvkm.h @@ -88,4 +88,7 @@ static inline int nvidia_vgpu_mgr_get_handle(struct pci_dev *pdev, #define nvidia_vgpu_mgr_bar1_unmap_mem(m, mem) \ m->handle.ops->bar1_unmap_mem(mem) +#define nvidia_vgpu_mgr_get_engine_bitmap(m, b) \ + m->handle.ops->get_engine_bitmap(m->handle.pf_drvdata, b) + #endif diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 124a1a4593ae..e06d5155bb38 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -7,6 +7,7 @@ #include #include +#include #include "vgpu_mgr.h" @@ -141,6 +142,91 @@ static int setup_mgmt_heap(struct nvidia_vgpu *vgpu) return 0; } +static int shutdown_vgpu_plugin_task(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + NV2080_CTRL_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK_PARAMS *ctrl; + + ctrl = nvidia_vgpu_mgr_rm_ctrl_get(vgpu_mgr, &vgpu->gsp_client, + NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_SHUTDOWN_GSP_VGPU_PLUGIN_TASK, + sizeof(*ctrl)); + if (IS_ERR(ctrl)) + return PTR_ERR(ctrl);; + + ctrl->gfid = vgpu->info.gfid; + + return nvidia_vgpu_mgr_rm_ctrl_wr(vgpu_mgr, &vgpu->gsp_client, + ctrl); +} + +static int cleanup_vgpu_plugin_task(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + NV2080_CTRL_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP_PARAMS *ctrl; + + ctrl = nvidia_vgpu_mgr_rm_ctrl_get(vgpu_mgr, &vgpu->gsp_client, + NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_VGPU_PLUGIN_CLEANUP, + sizeof(*ctrl)); + if (IS_ERR(ctrl)) + return PTR_ERR(ctrl); + + ctrl->gfid = vgpu->info.gfid; + + return nvidia_vgpu_mgr_rm_ctrl_wr(vgpu_mgr, &vgpu->gsp_client, + ctrl); +} + +static int bootload_vgpu_plugin_task(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + struct nvidia_vgpu_mgmt *mgmt = &vgpu->mgmt; + NV2080_CTRL_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK_PARAMS *ctrl; + DECLARE_BITMAP(engine_bitmap, NV2080_GPU_MAX_ENGINES); + int ret, i; + + ctrl = nvidia_vgpu_mgr_rm_ctrl_get(vgpu_mgr, &vgpu->gsp_client, + NV2080_CTRL_CMD_VGPU_MGR_INTERNAL_BOOTLOAD_GSP_VGPU_PLUGIN_TASK, + sizeof(*ctrl)); + if (IS_ERR(ctrl)) + return PTR_ERR(ctrl); + + ctrl->dbdf = vgpu->info.dbdf; + ctrl->gfid = vgpu->info.gfid; + ctrl->vmPid = 0; + ctrl->swizzId = 0; + ctrl->numChannels = vgpu->chid.num_chid; + ctrl->numPluginChannels = 0; + + bitmap_clear(engine_bitmap, 0, NV2080_GPU_MAX_ENGINES); + + /* FIXME: nvkm seems not correctly record engines. two engines are missing. */ + nvidia_vgpu_mgr_get_engine_bitmap(vgpu_mgr, engine_bitmap); + + for_each_set_bit(i, engine_bitmap, NV2080_GPU_MAX_ENGINES) + ctrl->chidOffset[i] = vgpu->chid.chid_offset; + + ctrl->bDisableDefaultSmcExecPartRestore = false; + ctrl->numGuestFbSegments = 1; + ctrl->guestFbPhysAddrList[0] = vgpu->fbmem_heap->addr; + ctrl->guestFbLengthList[0] = vgpu->fbmem_heap->size; + ctrl->pluginHeapMemoryPhysAddr = mgmt->heap_mem->addr; + ctrl->pluginHeapMemoryLength = mgmt->heap_mem->size; + ctrl->ctrlBuffOffset = 0; + ctrl->initTaskLogBuffOffset = mgmt->heap_mem->addr + + init_task_log_buff_offset(); + ctrl->initTaskLogBuffSize = init_task_log_buff_size(); + ctrl->vgpuTaskLogBuffOffset = ctrl->initTaskLogBuffOffset + + ctrl->initTaskLogBuffSize; + ctrl->vgpuTaskLogBuffSize = vgpu_task_log_buff_size(); + ctrl->bDeviceProfilingEnabled = false; + + ret = nvidia_vgpu_mgr_rm_ctrl_wr(vgpu_mgr, &vgpu->gsp_client, + ctrl); + if (ret) + return ret; + return 0; +} + /** * nvidia_vgpu_mgr_destroy_vgpu - destroy a vGPU instance * @vgpu: the vGPU instance going to be destroyed. @@ -154,6 +240,8 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + WARN_ON(shutdown_vgpu_plugin_task(vgpu)); + WARN_ON(cleanup_vgpu_plugin_task(vgpu)); nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu->gsp_client); clean_mgmt_heap(vgpu); clean_chids(vgpu); @@ -207,10 +295,16 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) goto err_alloc_gsp_client; + ret = bootload_vgpu_plugin_task(vgpu); + if (ret) + goto err_bootload_vgpu_plugin_task; + atomic_set(&vgpu->status, 1); return 0; +err_bootload_vgpu_plugin_task: + nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu->gsp_client); err_alloc_gsp_client: clean_mgmt_heap(vgpu); err_setup_mgmt_heap: From patchwork Sun Sep 22 12:49:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809117 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2077.outbound.protection.outlook.com [40.107.95.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16AC879DE for ; Sun, 22 Sep 2024 12:51:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009471; cv=fail; b=jTGz+3weMkHEJP+/GDeKaZKPpU2B3mWofnIjxwL3xlvp40JH5RXVTnMH5sIYb659QMYChEJ3Jiin65NBvuHxBtnOjvfQp4143G0528/TrfXDXH2LanZWsMrNqcYLBmzrLz2ZKWUT2L37L85scUnVvCFqx7SMv7WG87OGxmJLWRQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009471; c=relaxed/simple; bh=PIfzpcDhoynjzPHCW7yuWS0S2PkK1aRM4LUBT70tR+k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=HZRjc7f8kiWzy9FewFtl6SUhICp5KZxhZPF0rZmTUE9P07S3/+R5hZfvjJtUxtQIuVc49gdM6WdHmg1Ee1qZIguIZv2VtrcFGmSboPRLZE15hRbpGJ144WWvDrw4C7ocxTmr4+SMm94dsjZBh26jo71r9q3oQh64wzE3UC7VMt8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=NjzO66JU; arc=fail smtp.client-ip=40.107.95.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="NjzO66JU" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=pvJ7/sJKrISjXHaKEn1xQiLmP4jFdv8l1G+Do5/S0qTnWWiur2UgUjHJR6hnHReSqWnfOkinA3PhJ4Sj2TV0ClDHxInKawEC9AjU6K9pn+W/WD7N3nyBL/hMUgjKkAXRQku8G6mEagCQybHtT4RnVLPmYSSyVlKWzFfv8LU75SMliOU5MXmeEL3ZWJQLnHCJ5S3EwZEeiDiQ1ci4gMC1qcmFPjKVDwyD3BLhZ5/k+g1gKiWVcu9q/8fgZME1M740D+K/ZRt6dCkuNc3FBcgf04UgoayyTjwy/alBnS5ux6NmFaBAsG8CvRPOGXF4xMUxVRwWmv1G4yGEEfZgX/WKug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BleeluwThKtArWNEFfcTK34nznhGQWdAt359OAmj5hM=; b=pBdKjTzh+cdhqa3cQCu9SiHLSBLo7lkXuM30olG4Hpj1t0mrrD8+BzuOgJ94hTQ4xoCBN3jtJzn1LjsRMfqPSnbk1X1Wl4ZOo45TEGURs0CaDAgJS4b03YQzf9Xbuy08VsXfE5eeev214qFOyXAWYqJweA4k5kY7Yuyz6H8WVLQNEbY6TAe/xzvavKvWfF/mHSbjV8P8W0T84nlqx1QevNvFI6ozjuWrxM/C/+nw+EZ6eR78gnM9GQPAeiH6pXs4D0V3vwcuj5xxf94gP4sYPMhUpah/URbbpDZQRnKepwqWBIsU8+31ZkD5B/R5egnHzsrWk2teCZ8PSkNI6JxEzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BleeluwThKtArWNEFfcTK34nznhGQWdAt359OAmj5hM=; b=NjzO66JU53OZAiBvM6B/UrE6a+ApkHWAdWRxZB914PxEJNz9x2DEMydHU9cZfCrK8KL0ftyTY1mQjyjPkkIsCawMqJmVTIvXjFwXaXarBS23JLFHjOqyLkfE7RNC3rf2iX0+R2AxEO/MxwhwjQZVMGcrJ5D8Deq4MvrPcJRqWQMrx01WEov7tREeTzP9OTRNApHtZCc8FA+mm+QWbX8hM9CN6pjX+yez/iw8NhT0f0fVTq6wCfql0GIaufB3z97PJU2wUJA0hGYTzFrVO+H+WrYPfXQ47CPuB/kf+xjpEeb2NVOx5YAmjSwj+/AJa+nwzCLOEfYLW2cwoEDKiSWZmw== Received: from MN2PR11CA0027.namprd11.prod.outlook.com (2603:10b6:208:23b::32) by PH7PR12MB5655.namprd12.prod.outlook.com (2603:10b6:510:138::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.24; Sun, 22 Sep 2024 12:51:00 +0000 Received: from BL6PEPF00020E63.namprd04.prod.outlook.com (2603:10b6:208:23b:cafe::76) by MN2PR11CA0027.outlook.office365.com (2603:10b6:208:23b::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7939.29 via Frontend Transport; Sun, 22 Sep 2024 12:50:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E63.mail.protection.outlook.com (10.167.249.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:59 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:38 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:38 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:37 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 28/29] vfio/vgpu_mgr: introduce vGPU host RPC channel Date: Sun, 22 Sep 2024 05:49:50 -0700 Message-ID: <20240922124951.1946072-29-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E63:EE_|PH7PR12MB5655:EE_ X-MS-Office365-Filtering-Correlation-Id: bc7552e1-411c-44b5-0c59-08dcdb053556 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|36860700013|82310400026|1800799024; X-Microsoft-Antispam-Message-Info: =?utf-8?q?lxbFA9Ez1/cTE6vvmlaWEt3ivyoxyPK?= =?utf-8?q?vliqueVoR69izAKtA3lK3utXhG0hsVnS0sf3DV9nuJL3Pg8ioivwzKl6QZPgO6tiF?= =?utf-8?q?2LmhrWu2yneOBrSgQvV9BM9Mui6EVXGAHHSDVVvkELcOGEEXPU2hpZnkQFFMW3iuW?= =?utf-8?q?2fbuwPRfqW3wrR2oJyu5X6u1elzgPeDGehx2KayFDE4YXAZm3wBFDa6tAiplYsR4s?= =?utf-8?q?XA2jr8LCXCNGZ9uD0w+0Qpe3Qrl0ozlt1n6czvnhTGLL3XUX1Ryp82ZvBrs5rZApN?= =?utf-8?q?hFRuv2yzf8F1SyJ//Eh+pqrgrO40ce1FBtHEWM/wNhmqCH9Hwx3voqJLP7XmfkR7f?= =?utf-8?q?VOrFNL6IlPVbLobgvY5q3zahMb9njdncde4PNmuzhF/vewx1eS4LNWROSyA8pe9NC?= =?utf-8?q?EyK+GQX08qw6LuzfIzTvlYmB+LBwacOrcYRR910BK9vJx5cU/TpQihHIHZ6A0fVR3?= =?utf-8?q?kK3ycYVJhur8qX0PNoW8dQLO9CmoihX0uXdk1WoOPqxaTkUNcXOAxVJAOv49BpXzh?= =?utf-8?q?idQwxJwTOtI0jisQDdfJxmVEQNPwe1eGKdMhbpUYRcRzaRQ33OEZeXD1WOoHB+417?= =?utf-8?q?vSW9cqFdKiQOVVJo5A5Oj/SaUIXha0ldrsAjmDjPZlgD86aq2CoFonYi7XlPqVaBN?= =?utf-8?q?HXQduicEgM6c/KUYGjFP0PZYEgFU82h5IrTxwHF/+qlNCq+ytm5b8qnDaPvbbuJv6?= =?utf-8?q?nmVLkXKCioiIZ+KCYLeZyJ6Ujn0wTWOr08lfskbEGB17EDZWyii8Z1ukRu8Dd5Lb6?= =?utf-8?q?WKOAPLGdMTFeNsUc4/++52RbN5yNbo4kowO89Dd+FBXdMIgdhaCuvz70w4Han4QkY?= =?utf-8?q?Y4xO/sdtXNWVu9y2mWgtQ/+dh+STa3cLgV60WIEXdwXPpsFZfB2UAuYsQwO5EcAgr?= =?utf-8?q?a6HVj9RzsZKhMYKyvGNE4z9jiZ2pHIAJoTDiW11PDiNiQvqWjRzSSxxa/6Gxl55hE?= =?utf-8?q?EM/D8gFw5APHH0drLrT7/SDlryfPDJovHf/0K37EvAFVWa9HpL0p51dhIHSfRAZrR?= =?utf-8?q?uVZaDkz6lHwG4a5kXn+sdueK1Iq4btHug1pDoaNnpcjV7mrP8sturfyfQDkk/nbvy?= =?utf-8?q?I+r3zWEO+tCDWypHDdz5qSf585Iy68vACUf08zj50Dr9VXa8ar13alhDaxXgkra64?= =?utf-8?q?PFY95bxh36ysOJzrcgyQgIrRFizdR2AGojauzQqYbCqGXdttbOAfmlYgibonen5Fb?= =?utf-8?q?fTSp0XmF6a5tHmGNDsrk4gu4trfGig7jZ169xEc6vM6x8JkxMjQdzltSaUEPuI2Kt?= =?utf-8?q?UzfuPoqknXZOQ5Jo+wnha3ZjESnSkUvnW6LyFq5yL9qKF2fEmJYT8nIWKR4dwrjTz?= =?utf-8?q?yuRP8pE6T3zJ?= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(36860700013)(82310400026)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:59.3887 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bc7552e1-411c-44b5-0c59-08dcdb053556 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E63.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5655 A newly created vGPU requires some runtime configuration to be uploaded before moving on. Introduce the vGPU host RPCs manipulation APIs to send vGPU RPCs. Send vGPU RPCs to upload the runtime configuration of a vGPU. Cc: Neo Jia Cc: Surath Mitra Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/Makefile | 2 +- drivers/vfio/pci/nvidia-vgpu/debug.h | 18 ++ .../nvidia/inc/ctrl/ctrl0000/ctrl0000system.h | 30 +++ .../nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h | 213 +++++++++++++++ .../common/sdk/nvidia/inc/nv_vgpu_types.h | 51 ++++ .../common/sdk/vmioplugin/inc/vmioplugin.h | 26 ++ drivers/vfio/pci/nvidia-vgpu/rpc.c | 242 ++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 11 + drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c | 31 +++ drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 21 ++ 10 files changed, 644 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/nvidia-vgpu/debug.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000system.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/nv_vgpu_types.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/vmioplugin/inc/vmioplugin.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/rpc.c diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile index bd65fa548ea1..fade9d49df97 100644 --- a/drivers/vfio/pci/nvidia-vgpu/Makefile +++ b/drivers/vfio/pci/nvidia-vgpu/Makefile @@ -2,4 +2,4 @@ ccflags-y += -I$(srctree)/$(src)/include obj-$(CONFIG_NVIDIA_VGPU_MGR) += nvidia-vgpu-mgr.o -nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o vgpu_types.o +nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o vgpu_types.o rpc.o diff --git a/drivers/vfio/pci/nvidia-vgpu/debug.h b/drivers/vfio/pci/nvidia-vgpu/debug.h new file mode 100644 index 000000000000..bc1c4273f089 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/debug.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#ifndef __NVIDIA_VGPU_DEBUG_H__ +#define __NVIDIA_VGPU_DEBUG_H__ + +#define nv_vgpu_dbg(v, f, a...) \ + pci_dbg(v->pdev, "nvidia-vgpu %d: "f, v->info.id, ##a) + +#define nv_vgpu_info(v, f, a...) \ + pci_info(v->pdev, "nvidia-vgpu %d: "f, v->info.id, ##a) + +#define nv_vgpu_err(v, f, a...) \ + pci_err(v->pdev, "nvidia-vgpu %d: "f, v->info.id, ##a) + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000system.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000system.h new file mode 100644 index 000000000000..871c498fb666 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000system.h @@ -0,0 +1,30 @@ +#ifndef __src_common_sdk_nvidia_inc_ctrl_ctrl0000_ctrl0000system_h__ +#define __src_common_sdk_nvidia_inc_ctrl_ctrl0000_ctrl0000system_h__ + +/* Excerpt of RM headers from https://github.com/NVIDIA/open-gpu-kernel-modules/tree/535.113.01 */ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2005-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +#define NV0000_CTRL_CMD_SYSTEM_GET_VGX_SYSTEM_INFO_BUFFER_SIZE 256U +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h new file mode 100644 index 000000000000..8f3ea48ef10d --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/dev_vgpu_gsp.h @@ -0,0 +1,213 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ +#ifndef __src_common_sdk_nvidia_inc_vgpu_dev_nv_vgpu_gsp_h__ +#define __src_common_sdk_nvidia_inc_vgpu_dev_nv_vgpu_gsp_h__ + +#include "nv_vgpu_types.h" + +#define GSP_PLUGIN_BOOTLOADED 0x4E654A6F + +/******************************************************************************/ +/* GSP Control buffer shared between CPU Plugin and GSP Plugin - START */ +/******************************************************************************/ + +/* GSP Plugin heap memory layout + +--------------------------------+ offset = 0 + | CONTROL BUFFER | + +--------------------------------+ + | RESPONSE BUFFER | + +--------------------------------+ + | MESSAGE BUFFER | + +--------------------------------+ + | MIGRATION BUFFER | + +--------------------------------+ + | GSP PLUGIN ERROR BUFFER | + +--------------------------------+ + | INIT TASK LOG BUFFER | + +--------------------------------+ + | VGPU TASK LOG BUFFER | + +--------------------------------+ + | MEMORY AVAILABLE FOR | + | GSP PLUGIN INTERNAL HEAP USAGE | + +--------------------------------+ + */ +#define VGPU_CPU_GSP_CTRL_BUFF_VERSION 0x1 +#define VGPU_CPU_GSP_CTRL_BUFF_REGION_SIZE 4096 +#define VGPU_CPU_GSP_RESPONSE_BUFF_REGION_SIZE 4096 +#define VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE 4096 +#define VGPU_CPU_GSP_MIGRATION_BUFF_REGION_SIZE (2 * 1024 * 1024) +#define VGPU_CPU_GSP_ERROR_BUFF_REGION_SIZE 4096 +#define VGPU_CPU_GSP_INIT_TASK_LOG_BUFF_REGION_SIZE (128 * 1024) +#define VGPU_CPU_GSP_VGPU_TASK_LOG_BUFF_REGION_SIZE (256 * 1024) +#define VGPU_CPU_GSP_COMMUNICATION_BUFF_TOTAL_SIZE (VGPU_CPU_GSP_CTRL_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_RESPONSE_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_MIGRATION_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_ERROR_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_INIT_TASK_LOG_BUFF_REGION_SIZE + \ + VGPU_CPU_GSP_VGPU_TASK_LOG_BUFF_REGION_SIZE) + +// +// Control buffer: CPU Plugin -> GSP Plugin +// CPU Plugin - Write only +// GSP Plugin - Read only +// +typedef union { + NvU8 buf[VGPU_CPU_GSP_CTRL_BUFF_REGION_SIZE]; + struct { + volatile NvU32 version; // Version of format + volatile NvU32 message_type; // Task to be performed by GSP Plugin + volatile NvU32 message_seq_num; // Incrementing sequence number to identify the RPC packet + volatile NvU64 response_buff_offset; // Buffer used to send data from GSP Plugin -> CPU Plugin + volatile NvU64 message_buff_offset; // Buffer used to send RPC data between CPU and GSP Plugin + volatile NvU64 migration_buff_offset; // Buffer used to send migration data between CPU and GSP Plugin + volatile NvU64 error_buff_offset; // Buffer used to send error data from GSP Plugin -> CPU Plugin + volatile NvU32 migration_buf_cpu_access_offset;// CPU plugin GET/PUT offset of migration buffer + volatile NvBool is_migration_in_progress; // Is migration active or cancelled + volatile NvU32 error_buff_cpu_get_idx; // GET pointer into ERROR Buffer for CPU Plugin + volatile NvU32 attached_vgpu_count; + volatile struct { + NvU32 vgpu_type_id; + NvU32 host_gpu_pci_id; + NvU32 pci_dev_id; + NvU8 vgpu_uuid[VM_UUID_SIZE]; + } host_info[VMIOPD_MAX_INSTANCES]; + }; +} VGPU_CPU_GSP_CTRL_BUFF_REGION; + +// +// Specify actions intended on getting +// notification from CPU Plugin -> GSP plugin +// +typedef enum { + NV_VGPU_CPU_RPC_MSG_VERSION_NEGOTIATION = 1, + NV_VGPU_CPU_RPC_MSG_SETUP_CONFIG_PARAMS_AND_INIT, + NV_VGPU_CPU_RPC_MSG_RESET, + NV_VGPU_CPU_RPC_MSG_MIGRATION_STOP_WORK, + NV_VGPU_CPU_RPC_MSG_MIGRATION_CANCEL_STOP, + NV_VGPU_CPU_RPC_MSG_MIGRATION_SAVE_STATE, + NV_VGPU_CPU_RPC_MSG_MIGRATION_CANCEL_SAVE, + NV_VGPU_CPU_RPC_MSG_MIGRATION_RESTORE_STATE, + NV_VGPU_CPU_RPC_MSG_MIGRATION_RESTORE_DEFERRED_STATE, + NV_VGPU_CPU_RPC_MSG_MIGRATION_RESUME_WORK, + NV_VGPU_CPU_RPC_MSG_CONSOLE_VNC_STATE, + NV_VGPU_CPU_RPC_MSG_VF_BAR0_REG_ACCESS, + NV_VGPU_CPU_RPC_MSG_UPDATE_BME_STATE, + NV_VGPU_CPU_RPC_MSG_GET_GUEST_INFO, + NV_VGPU_CPU_RPC_MSG_MAX, +} MESSAGE; + +// +// Params structure for NV_VGPU_CPU_RPC_MSG_VERSION_NEGOTIATION +// +typedef struct { + volatile NvU32 version_cpu; /* Sent by CPU Plugin */ + volatile NvU32 version_negotiated; /* Updated by GSP Plugin */ +} NV_VGPU_CPU_RPC_DATA_VERSION_NEGOTIATION; + +// +// Host CPU arch +// +typedef enum { + NV_VGPU_HOST_CPU_ARCH_AARCH64 = 1, + NV_VGPU_HOST_CPU_ARCH_X86_64, +} NV_VGPU_HOST_CPU_ARCH; + +// +// Params structure for NV_VGPU_CPU_RPC_MSG_COPY_CONFIG_PARAMS +// +typedef struct { + volatile NvU8 vgpu_uuid[VM_UUID_SIZE]; + volatile NvU32 dbdf; + volatile NvU32 driver_vm_vf_dbdf; + volatile NvU32 vgpu_device_instance_id; + volatile NvU32 vgpu_type; + volatile NvU32 vm_pid; + volatile NvU32 swizz_id; + volatile NvU32 num_channels; + volatile NvU32 num_plugin_channels; + volatile NvU32 vmm_cap; + volatile NvU32 migration_feature; + volatile NvU32 hypervisor_type; + volatile NvU32 host_cpu_arch; + volatile NvU64 host_page_size; + volatile NvBool rev1[2]; + volatile NvBool enable_uvm; + volatile NvBool linux_interrupt_optimization; + volatile NvBool vmm_migration_supported; + volatile NvBool rev2; + volatile NvBool enable_console_vnc; + volatile NvBool use_non_stall_linux_events; + volatile NvU32 rev3; +} NV_VGPU_CPU_RPC_DATA_COPY_CONFIG_PARAMS; + +// Params structure for NV_VGPU_CPU_RPC_MSG_UPDATE_BME_STATE +typedef struct { + volatile NvBool enable; + volatile NvBool allowed; +} NV_VGPU_CPU_RPC_DATA_UPDATE_BME_STATE; +// +// Message Buffer: +// CPU Plugin - Read/Write +// GSP Plugin - Read/Write +// +typedef union { + NvU8 buf[VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE]; + NV_VGPU_CPU_RPC_DATA_VERSION_NEGOTIATION version_data; + NV_VGPU_CPU_RPC_DATA_COPY_CONFIG_PARAMS config_data; + NV_VGPU_CPU_RPC_DATA_UPDATE_BME_STATE bme_state; +} VGPU_CPU_GSP_MSG_BUFF_REGION; + +typedef struct { + volatile NvU64 sequence_update_start; + volatile NvU64 sequence_update_end; + volatile NvU32 effective_fb_page_size; + volatile NvU32 rect_width; + volatile NvU32 rect_height; + volatile NvU32 surface_width; + volatile NvU32 surface_height; + volatile NvU32 surface_size; + volatile NvU32 surface_offset; + volatile NvU32 surface_format; + volatile NvU32 surface_kind; + volatile NvU32 surface_pitch; + volatile NvU32 surface_type; + volatile NvU8 surface_block_height; + volatile vmiop_bool_t is_blanking_enabled; + volatile vmiop_bool_t is_flip_pending; + volatile vmiop_bool_t is_free_pending; + volatile vmiop_bool_t is_memory_blocklinear; +} VGPU_CPU_GSP_DISPLAYLESS_SURFACE; + +// +// GSP Plugin Response Buffer: +// CPU Plugin - Read only +// GSP Plugin - Write only +// +typedef union { + NvU8 buf[VGPU_CPU_GSP_RESPONSE_BUFF_REGION_SIZE]; + struct { + // Updated by GSP Plugin once task is complete + volatile NvU32 message_seq_num_processed; + // Updated by GSP on completion of RPC + volatile NvU32 result_code; + volatile NvU32 guest_rpc_version; + // GSP plugin GET/PUT offset pointer of migration buffer + volatile NvU32 migration_buf_gsp_access_offset; + // Current state of migration + volatile NvU32 migration_state_save_complete; + // Console VNC surface information + volatile VGPU_CPU_GSP_DISPLAYLESS_SURFACE surface[VMIOPD_MAX_HEADS]; + // PUT pointer into ERROR Buffer for GSP Plugin + volatile NvU32 error_buff_gsp_put_idx; + // Updated grid license state as received from guest + volatile NvU32 grid_license_state; + }; +} VGPU_CPU_GSP_RESPONSE_BUFF_REGION; + +/******************************************************************************/ +/* GSP Control buffer shared between CPU Plugin and GSP Plugin - END */ +/******************************************************************************/ +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/nv_vgpu_types.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/nv_vgpu_types.h new file mode 100644 index 000000000000..903a5840366c --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/nvidia/inc/nv_vgpu_types.h @@ -0,0 +1,51 @@ +#ifndef __src_common_sdk_nvidia_inc_nv_vgpu_types_h__ +#define __src_common_sdk_nvidia_inc_nv_vgpu_types_h__ + +/* Excerpt of RM headers from https://github.com/NVIDIA/open-gpu-kernel-modules/tree/535.113.01 */ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2016-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: MIT + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + */ + +#define VM_UUID_SIZE 16 +#define INVALID_VGPU_DEV_INST 0xFFFFFFFFU +#define MAX_VGPU_DEVICES_PER_VM 16U + +/* This enum represents the current state of guest dependent fields */ +typedef enum GUEST_VM_INFO_STATE { + GUEST_VM_INFO_STATE_UNINITIALIZED = 0, + GUEST_VM_INFO_STATE_INITIALIZED = 1, +} GUEST_VM_INFO_STATE; + +/* This enum represents types of VM identifiers */ +typedef enum VM_ID_TYPE { + VM_ID_DOMAIN_ID = 0, + VM_ID_UUID = 1, +} VM_ID_TYPE; + +/* This structure represents VM identifier */ +typedef union VM_ID { + NvU8 vmUuid[VM_UUID_SIZE]; + NV_DECLARE_ALIGNED(NvU64 vmId, 8); +} VM_ID; + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/vmioplugin/inc/vmioplugin.h b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/vmioplugin/inc/vmioplugin.h new file mode 100644 index 000000000000..58a473309e42 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/include/nvrm/common/sdk/vmioplugin/inc/vmioplugin.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ +#ifndef __src_common_sdk_vmioplugin_inc_vmioplugin_h__ +#define __src_common_sdk_vmioplugin_inc_vmioplugin_h__ + +#define VMIOPD_MAX_INSTANCES 16 +#define VMIOPD_MAX_HEADS 4 + +/** + * Boolean type. + */ + +enum vmiop_bool_e { + vmiop_false = 0, /*!< Boolean false */ + vmiop_true = 1 /*!< Boolean true */ +}; + +/** + * Boolean type. + */ + +typedef enum vmiop_bool_e vmiop_bool_t; + +#endif diff --git a/drivers/vfio/pci/nvidia-vgpu/rpc.c b/drivers/vfio/pci/nvidia-vgpu/rpc.c new file mode 100644 index 000000000000..c316941f4b97 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/rpc.c @@ -0,0 +1,242 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include +#include + +#include +#include +#include +#include +#include + +#include "debug.h" +#include "vgpu_mgr.h" + +static void trigger_doorbell(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + + u32 v = vgpu->info.gfid * 32 + 17; + + writel(v, vgpu_mgr->bar0_vaddr + 0x00B80000 + 0x2200); + readl(vgpu_mgr->bar0_vaddr + 0x00B80000 + 0x2200); +} + +static void send_rpc_request(struct nvidia_vgpu *vgpu, u32 msg_type, + void *data, u64 size) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + VGPU_CPU_GSP_CTRL_BUFF_REGION *ctrl_buf = rpc->ctrl_buf; + + if (data && size) + memcpy_toio(rpc->msg_buf, data, size); + + ctrl_buf->message_type = msg_type; + + rpc->msg_seq_num++; + ctrl_buf->message_seq_num = rpc->msg_seq_num; + + trigger_doorbell(vgpu); +} + +static int wait_for_response(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + VGPU_CPU_GSP_RESPONSE_BUFF_REGION *resp_buf = rpc->resp_buf; + + u64 timeout = 120 * 1000000; /* 120s */ + + do { + if (resp_buf->message_seq_num_processed == rpc->msg_seq_num) + break; + + usleep_range(1, 2); + } while (--timeout); + + return timeout ? 0 : -ETIMEDOUT; +} + +static int recv_rpc_response(struct nvidia_vgpu *vgpu, void *data, + u64 size, u32 *result) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + VGPU_CPU_GSP_RESPONSE_BUFF_REGION *resp_buf = rpc->resp_buf; + int ret; + + ret = wait_for_response(vgpu); + if (result) + *result = resp_buf->result_code; + + if (ret) + return ret; + + if (data && size) + memcpy_fromio(data, rpc->msg_buf, size); + + return 0; +} + +int nvidia_vgpu_rpc_call(struct nvidia_vgpu *vgpu, u32 msg_type, + void *data, u64 size) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + u32 result; + int ret; + + if (WARN_ON(msg_type >= NV_VGPU_CPU_RPC_MSG_MAX) || + (size > VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE) || + ((size != 0) && (data == NULL))) + return -EINVAL; + + mutex_lock(&rpc->lock); + + send_rpc_request(vgpu, msg_type, data, size); + ret = recv_rpc_response(vgpu, data, size, &result); + + mutex_unlock(&rpc->lock); + if (ret || result) { + nv_vgpu_err(vgpu, "fail to recv RPC: result %u\n", + result); + return -EINVAL; + } + return ret; +} + +void nvidia_vgpu_clean_rpc(struct nvidia_vgpu *vgpu) +{ +} + +static void init_rpc_buf_pointers(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgmt *mgmt = &vgpu->mgmt; + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + + rpc->ctrl_buf = mgmt->ctrl_vaddr; + rpc->resp_buf = rpc->ctrl_buf + VGPU_CPU_GSP_CTRL_BUFF_REGION_SIZE; + rpc->msg_buf = rpc->resp_buf + VGPU_CPU_GSP_RESPONSE_BUFF_REGION_SIZE; + rpc->migration_buf = rpc->msg_buf + VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE; + rpc->error_buf = rpc->migration_buf + VGPU_CPU_GSP_MIGRATION_BUFF_REGION_SIZE; +} + +static void init_ctrl_buf_offsets(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + VGPU_CPU_GSP_CTRL_BUFF_REGION *ctrl_buf; + u64 offset = 0; + + ctrl_buf = rpc->ctrl_buf; + + ctrl_buf->version = VGPU_CPU_GSP_CTRL_BUFF_VERSION; + + offset = VGPU_CPU_GSP_CTRL_BUFF_REGION_SIZE; + ctrl_buf->response_buff_offset = offset; + + offset += VGPU_CPU_GSP_RESPONSE_BUFF_REGION_SIZE; + ctrl_buf->message_buff_offset = offset; + + offset += VGPU_CPU_GSP_MESSAGE_BUFF_REGION_SIZE; + ctrl_buf->migration_buff_offset = offset; + + offset += VGPU_CPU_GSP_MIGRATION_BUFF_REGION_SIZE; + ctrl_buf->error_buff_offset = offset; +} + +static int wait_vgpu_plugin_task_bootloaded(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + VGPU_CPU_GSP_CTRL_BUFF_REGION *ctrl_buf = rpc->ctrl_buf; + + u64 timeout = 10 * 1000000; /* 10 s */ + + do { + if (ctrl_buf->message_seq_num == GSP_PLUGIN_BOOTLOADED) + break; + + usleep_range(1, 2); + } while (--timeout); + + return timeout ? 0 : -ETIMEDOUT; +} + +static int negotiate_rpc_version(struct nvidia_vgpu *vgpu) +{ + return nvidia_vgpu_rpc_call(vgpu, NV_VGPU_CPU_RPC_MSG_VERSION_NEGOTIATION, + NULL, 0); +} + +unsigned char config_params[] = { + 0x24, 0xef, 0x8f, 0xf7, 0x3e, 0xd5, 0x11, 0xef, 0xae, 0x36, 0x97, 0x58, + 0xb1, 0xcb, 0x0c, 0x87, 0x04, 0xc1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x14, 0x00, 0xd0, 0xc1, 0x65, 0x03, 0x00, 0x00, 0xa1, 0x0e, 0x00, 0x00, + 0xff, 0xff, 0xff, 0xff, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, + 0x02, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x01, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00 +}; + +static int send_config_params_and_init(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = vgpu->vgpu_mgr; + NV_VGPU_CPU_RPC_DATA_COPY_CONFIG_PARAMS params = {0}; + NVA081_CTRL_VGPU_INFO *info = (NVA081_CTRL_VGPU_INFO *) + vgpu->vgpu_type; + + memcpy(¶ms, config_params, sizeof(config_params)); + + params.dbdf = vgpu->info.dbdf; + params.vgpu_device_instance_id = + nvidia_vgpu_mgr_get_gsp_client_handle(vgpu_mgr, &vgpu->gsp_client); + params.vgpu_type = info->vgpuType; + params.vm_pid = 0; + params.swizz_id = 0; + params.num_channels = vgpu->chid.num_chid; + params.num_plugin_channels = vgpu->chid.num_plugin_channels; + + return nvidia_vgpu_rpc_call(vgpu, NV_VGPU_CPU_RPC_MSG_SETUP_CONFIG_PARAMS_AND_INIT, + ¶ms, sizeof(params)); +} + +/** + * nvidia_vgpu_setup_rpc - setup the vGPU host RPC channel and send runtime + * configuration. + * @vgpu: the vGPU instance. + * @vgpu_type: the vGPU type of the vGPU instance. + * + * Returns: 0 on success, others on failure. + */ +int nvidia_vgpu_setup_rpc(struct nvidia_vgpu *vgpu) +{ + struct nvidia_vgpu_rpc *rpc = &vgpu->rpc; + int ret; + + mutex_init(&rpc->lock); + + init_rpc_buf_pointers(vgpu); + init_ctrl_buf_offsets(vgpu); + + ret = wait_vgpu_plugin_task_bootloaded(vgpu); + if (ret) { + nv_vgpu_err(vgpu, "waiting bootload timeout!\n"); + return ret; + } + + ret = negotiate_rpc_version(vgpu); + if (ret) { + nv_vgpu_err(vgpu, "fail to negotiate rpc version!\n"); + return ret; + } + + ret = send_config_params_and_init(vgpu); + if (ret) { + nv_vgpu_err(vgpu, "fail to init vgpu plugin task!\n"); + return ret; + } + + nv_vgpu_dbg(vgpu, "vGPU RPC initialization is done.\n"); + + return 0; +} diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index e06d5155bb38..93d27db30a41 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -8,6 +8,9 @@ #include #include #include +#include +#include +#include #include "vgpu_mgr.h" @@ -240,6 +243,7 @@ int nvidia_vgpu_mgr_destroy_vgpu(struct nvidia_vgpu *vgpu) if (!atomic_cmpxchg(&vgpu->status, 1, 0)) return -ENODEV; + nvidia_vgpu_clean_rpc(vgpu); WARN_ON(shutdown_vgpu_plugin_task(vgpu)); WARN_ON(cleanup_vgpu_plugin_task(vgpu)); nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu->gsp_client); @@ -299,10 +303,17 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) if (ret) goto err_bootload_vgpu_plugin_task; + ret = nvidia_vgpu_setup_rpc(vgpu); + if (ret) + goto err_setup_rpc; + atomic_set(&vgpu->status, 1); return 0; +err_setup_rpc: + shutdown_vgpu_plugin_task(vgpu); + cleanup_vgpu_plugin_task(vgpu); err_bootload_vgpu_plugin_task: nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu->gsp_client); err_alloc_gsp_client: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c index dcb314b14f91..e84cf4a845d4 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.c @@ -7,11 +7,35 @@ DEFINE_MUTEX(vgpu_mgr_attach_lock); +static void unmap_pf_mmio(struct nvidia_vgpu_mgr *vgpu_mgr) +{ + iounmap(vgpu_mgr->bar0_vaddr); +} + +static int map_pf_mmio(struct nvidia_vgpu_mgr *vgpu_mgr) +{ + struct pci_dev *pdev = vgpu_mgr->pdev; + resource_size_t start, size; + void *vaddr; + + start = pci_resource_start(pdev, 0); + size = pci_resource_len(pdev, 0); + + vaddr = ioremap(start, size); + if (!vaddr) + return -ENOMEM; + + vgpu_mgr->bar0_vaddr = vaddr; + + return 0; +} + static void vgpu_mgr_release(struct kref *kref) { struct nvidia_vgpu_mgr *vgpu_mgr = container_of(kref, struct nvidia_vgpu_mgr, refcount); + unmap_pf_mmio(vgpu_mgr); nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu_mgr->gsp_client); nvidia_vgpu_mgr_detach_handle(&vgpu_mgr->handle); kvfree(vgpu_mgr); @@ -83,6 +107,8 @@ struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev) kref_init(&vgpu_mgr->refcount); mutex_init(&vgpu_mgr->vgpu_id_lock); + vgpu_mgr->pdev = dev->physfn; + ret = nvidia_vgpu_mgr_alloc_gsp_client(vgpu_mgr, &vgpu_mgr->gsp_client); if (ret) @@ -92,9 +118,14 @@ struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev) if (ret) goto fail_init_vgpu_types; + ret = map_pf_mmio(vgpu_mgr); + if (ret) + goto fail_map_pf_mmio; + mutex_unlock(&vgpu_mgr_attach_lock); return vgpu_mgr; +fail_map_pf_mmio: fail_init_vgpu_types: nvidia_vgpu_mgr_free_gsp_client(vgpu_mgr, &vgpu_mgr->gsp_client); fail_alloc_gsp_client: diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index 6f05b285484c..af922d8e539c 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -28,6 +28,16 @@ struct nvidia_vgpu_mgmt { void __iomem *vgpu_task_log_vaddr; }; +struct nvidia_vgpu_rpc { + struct mutex lock; + u32 msg_seq_num; + void __iomem *ctrl_buf; + void __iomem *resp_buf; + void __iomem *msg_buf; + void __iomem *migration_buf; + void __iomem *error_buf; +}; + struct nvidia_vgpu { struct mutex lock; atomic_t status; @@ -41,6 +51,7 @@ struct nvidia_vgpu { struct nvidia_vgpu_mem *fbmem_heap; struct nvidia_vgpu_chid chid; struct nvidia_vgpu_mgmt mgmt; + struct nvidia_vgpu_rpc rpc; }; struct nvidia_vgpu_mgr { @@ -55,6 +66,9 @@ struct nvidia_vgpu_mgr { u32 num_vgpu_types; struct nvidia_vgpu_gsp_client gsp_client; + + struct pci_dev *pdev; + void __iomem *bar0_vaddr; }; struct nvidia_vgpu_mgr *nvidia_vgpu_mgr_get(struct pci_dev *dev); @@ -65,4 +79,11 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type); int nvidia_vgpu_mgr_init_vgpu_types(struct nvidia_vgpu_mgr *vgpu_mgr); +int nvidia_vgpu_rpc_call(struct nvidia_vgpu *vgpu, u32 msg_type, + void *data, u64 size); +void nvidia_vgpu_clean_rpc(struct nvidia_vgpu *vgpu); +int nvidia_vgpu_setup_rpc(struct nvidia_vgpu *vgpu); + +int nvidia_vgpu_mgr_reset_vgpu(struct nvidia_vgpu *vgpu); + #endif From patchwork Sun Sep 22 12:49:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809116 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2081.outbound.protection.outlook.com [40.107.96.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B82B7489 for ; Sun, 22 Sep 2024 12:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009471; cv=fail; b=dAUlfhEot1nbfQf2yFjzcM7KjIp3HoxlwbchNMCETXcK2C1vlqevrQYKfulKODbAWjiikGCNqluwHPOEHeuOHCB3a/Ri7mH9rGKEqBzB/JoaytvSYKBpDBibReFEivxTrAq8KMHox0oPjLD0kgTV7fSo6z2DLzwETciVJuSZg/o= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009471; c=relaxed/simple; bh=mWAur9sYpWSZbsDXaS3F0dCp4UHonddEFIfCRRlDA94=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SjWkrz411BX8ItIPlwBFq98K+lyqhTVBja0E1bm3H8QN8pMcV/s+rPkon0V5AcFOoqI6+37F/rQG0LpHcalHhOO/DkX14FkhsOvWc6KDc09XSGBBctK20B8VAY1O5wjOsnpeKUaqpdZnIXeBg92U12lsgxP6KLzqbZSflBfe64I= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GUrlyS4v; arc=fail smtp.client-ip=40.107.96.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GUrlyS4v" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xTo6ImueNYBucZQCFnLHET9JC14YiFjcXsQuiQJDxqz6ZCH1IBbs8DlR5WiWEAXMXRstIN6JL2dxyGVE/YBdsAhIO/HLbSXxiYpppYhKbzHIrrwHwSl2A9q58WG8PxBHZuok4APkGYJpiiA0MJK/clSjLiOl+SeYtkODHSZwDVWTrGYmwiyxLXli9CAUSTojNKkt6bOFpdsXcwzc6CYCKWwwUVJZoO4gtQO6CkjguFLsgkrpm/Jk2pkRZzVvk1W6y9x7eCa8tOSygjO89taoTTGpH7nB7r7vyeiDZBZhMO4eXnHoCF0zDd/XKJP+/63NeniBEA916gGGEdPDyMhOXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2jVCiig4wQkEsHJ2djmxlaLFeWRiIf4rWWQ+EMJrKiQ=; b=iEfEfaESq+EsNvfdUfXarx26c/ieii25IkXL/fKBBAXEe39kIU40bOe2YtinMLj1YckCwAE+pxlRxXiXuW51NIym5K9pzq5kR0c8a4ccTe3EEg63Gw8lSXKtWHT9oxsAr7GSnBsj6HejkvHfsCCumjbaTzDdrRPidCub8HZJy/cIdFbCnzJL2hbbO+IUeI6tmrjSd/wx9PJBYo19tCI22h2T+z/W3gPZxVPVRufVAxo0rqqqTdVuvhf5eGKlKCqCAqzKGn0rCs8PxtPv3YSR0ZsEvQr9BjfvaAxjQxVFCfxlA0DtzFSpfrVriX+rHiGhsIYGjVnHDV7morcIKde/wQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2jVCiig4wQkEsHJ2djmxlaLFeWRiIf4rWWQ+EMJrKiQ=; b=GUrlyS4v/+GXrf7gHEE/ViRCANxvCmR9dQOpT27CuhcC8zDzKaC4NXNVwCnJRh0/jvnz9W9sl2947GuDaADnNtRPy3zf+u1wVjos3J4UYWwbcV6V9yfL/eo0ZJyZQ2ET8nFEkMvpj6+ul0YhpI2Le8xc7fySEb3WizH+JyJCRrDO5AgfmHl3YxeZkV55LvmrNBYZnWM6jEK9D/uJGSfeV0eepZhT4/INN/PjoDBNlPL6V8ss4QC/vu83RJdgwoYE5xzM4rbjd0Ny2gvne3Isu08KKnf4tMD3MGupsbmoI9BVaAzUf0BOCrmJWfjpQUjeu6kI3jZIW58TFlHoZ2MtlQ== Received: from MW4P221CA0026.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::31) by BY5PR12MB4195.namprd12.prod.outlook.com (2603:10b6:a03:200::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:59 +0000 Received: from CO1PEPF000075F1.namprd03.prod.outlook.com (2603:10b6:303:8b:cafe::f7) by MW4P221CA0026.outlook.office365.com (2603:10b6:303:8b::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.25 via Frontend Transport; Sun, 22 Sep 2024 12:50:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by CO1PEPF000075F1.mail.protection.outlook.com (10.167.249.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:58 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:39 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:38 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:38 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , , Vinay Kabra Subject: [RFC 29/29] vfio/vgpu_mgr: introduce NVIDIA vGPU VFIO variant driver Date: Sun, 22 Sep 2024 05:49:51 -0700 Message-ID: <20240922124951.1946072-30-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000075F1:EE_|BY5PR12MB4195:EE_ X-MS-Office365-Filtering-Correlation-Id: 48f60ba1-526d-4e70-c890-08dcdb0534e4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026; X-Microsoft-Antispam-Message-Info: =?utf-8?q?2WM3UnEUEn33J4EEjJWnWuIc6EhrtVA?= =?utf-8?q?SFT7hXkgXnIajhJYXXYwlaWRCX1CjWIVq3j6yUOAs05aeSgIn82c+sx5r2zzNMSud?= =?utf-8?q?/bgZzPQR9xbujwcgZ4FefXWunMNJud9XnUgB4WoRi+hBG5Z8D7TgrTw25RnaSolpq?= =?utf-8?q?NMfpOEghylm/8/iU83mJM1E4L5LW9kxw5d+qEv81GlHnW9tM54lMRXa2/YOHxhm/f?= =?utf-8?q?mszgyGl8NCp/rXYFZ6qrc7/m19Vw1PbRh8hUzYbrgEXmxFN5e+EMoEB3r96kbm7/0?= =?utf-8?q?odBKIkYTqywTAG+kTteR7UgNKyWEWkSvzC7sGw9M1OBUu+zmrxwiUGE2EU+coSF7C?= =?utf-8?q?VuoAAq6LLErBgfGiz41n/GoeylgE1n83MsvwQmq/WjUF+oWoBp8FaLyrGleN1qilE?= =?utf-8?q?XhsTQEbybGYs4ByJ3Su7hjfPE90l39XaNjTplP4qe/HfXZMrkw43xezfQ2IdIqm1N?= =?utf-8?q?2qfKg93RY2oIOys6c930Danmlsrs6jKipf39HqMI63YOXxhafjrRG1RLWUO7uunSe?= =?utf-8?q?p0YEj03TfUbBhafILTsVsxBD0h0nJ+iecFwbBvlz+LtFiLvJMRWZZZQk9JGku3yCH?= =?utf-8?q?SO6UFrwjN209zE0J1Lae+UEcCIK1eskUW0k2ax7wFiqbyY10dM0qzlR0UAGKWzS8D?= =?utf-8?q?rQxXI9ICK4C06FnAjEpAFKTY4VRdRlu5LQ5w+j1IRdsgGupMO2mMm1EkvMTkVhPnr?= =?utf-8?q?wE87YZCp40Nt6K9AFujG3d6UCSQY02IRBKIf1inhhhbJzSqMliZyMg7ntZ4dCt7fv?= =?utf-8?q?HMkUFbxbhPRrfINgmJjbYYPE6fAVng1eKKAcNJZn0JGoJ/M0oNWgFK0X0b7eDhdBQ?= =?utf-8?q?U/ZYw4ecNIraOIw63FBg3SwvhbupSu4GPWdC7bOTd5kM1w5GiY+j/N+opXUUiofWv?= =?utf-8?q?nTq+I48QSnP4Eg9g/Zt/RG8ENbY+x3DCJLJFEhE4l4BNwA+4Ch/+BuGY+dnxnz8N6?= =?utf-8?q?/MPaiLzCclDerP/ss2DDoSwruYNZcLjIyPdeaKEbiLUU1S5PfyJWx6YwbFaqzCtJW?= =?utf-8?q?9vowO4yjG2wQF8/tqjh+U19eOcYEmtRiUJAxJq89UcDSQbq3EUoiGYrXzpzYnI1KC?= =?utf-8?q?lqISxVfU+opAz7UW+KHqcWaLtfonyWet5ydy7aUvVJxeRBsd7qdsWPMtKZ8Ebl5oI?= =?utf-8?q?CcAdBstWcAtS0pw5J17Fhjs9QqQ81Madk2XThOgGCVYT+mQWu5tQUHg7TZJhIyAU5?= =?utf-8?q?LySxtXd0gWnrSn95lacNCoc2wH9wbXOSGTr3P5RXyWIPixOBgBtqogU7hXa2Gc0/B?= =?utf-8?q?nycmaEPB1Kmkb8wRao6zXx2RcFI8g2oZXGhv6qTkwtK/aHamdzGtLqfSyAfqyLxtU?= =?utf-8?q?SFX6M1hFOr9l?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:58.7714 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 48f60ba1-526d-4e70-c890-08dcdb0534e4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000075F1.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4195 A VFIO variant driver module is designed to extend the capabilities of the existing VFIO (Virtual Function I/O), offering device management interfaces to the userspace and advanced feature support. For the userspace to use the NVIDIA vGPU, a new vGPU VFIO variant driver is introduced to provide vGPU management, like selecting/creating vGPU instance, support advance features like live migration. Introduce the NVIDIA vGPU VFIO variant driver to support vGPU lifecycle management UABI and the future advancd features. Cc: Neo Jia Cc: Surath Mitra Cc: Kirti Wankhede Cc: Vinay Kabra Cc: Ankit Agrawal Signed-off-by: Zhi Wang --- drivers/vfio/pci/nvidia-vgpu/Makefile | 3 + drivers/vfio/pci/nvidia-vgpu/vfio.h | 43 ++ drivers/vfio/pci/nvidia-vgpu/vfio_access.c | 297 ++++++++++++ drivers/vfio/pci/nvidia-vgpu/vfio_main.c | 511 +++++++++++++++++++++ drivers/vfio/pci/nvidia-vgpu/vgpu.c | 22 + drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h | 2 +- 6 files changed, 877 insertions(+), 1 deletion(-) create mode 100644 drivers/vfio/pci/nvidia-vgpu/vfio.h create mode 100644 drivers/vfio/pci/nvidia-vgpu/vfio_access.c create mode 100644 drivers/vfio/pci/nvidia-vgpu/vfio_main.c diff --git a/drivers/vfio/pci/nvidia-vgpu/Makefile b/drivers/vfio/pci/nvidia-vgpu/Makefile index fade9d49df97..99c47e2f436d 100644 --- a/drivers/vfio/pci/nvidia-vgpu/Makefile +++ b/drivers/vfio/pci/nvidia-vgpu/Makefile @@ -3,3 +3,6 @@ ccflags-y += -I$(srctree)/$(src)/include obj-$(CONFIG_NVIDIA_VGPU_MGR) += nvidia-vgpu-mgr.o nvidia-vgpu-mgr-y := vgpu_mgr.o vgpu.o vgpu_types.o rpc.o + +obj-$(CONFIG_NVIDIA_VGPU_VFIO_PCI) += nvidia-vgpu-vfio-pci.o +nvidia-vgpu-vfio-pci-y := vfio_main.o vfio_access.o diff --git a/drivers/vfio/pci/nvidia-vgpu/vfio.h b/drivers/vfio/pci/nvidia-vgpu/vfio.h new file mode 100644 index 000000000000..fa6bbf81552d --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vfio.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#ifndef _NVIDIA_VGPU_VFIO_H__ +#define _NVIDIA_VGPU_VFIO_H__ + +#include + +#include +#include +#include + +#include "vgpu_mgr.h" + +#define VGPU_CONFIG_PARAMS_MAX_LENGTH 1024 +#define DEVICE_CLASS_LENGTH 5 +#define PCI_CONFIG_SPACE_LENGTH 4096 + +#define CAP_LIST_NEXT_PTR_MSIX 0x7c +#define MSIX_CAP_SIZE 0xc + +struct nvidia_vgpu_vfio { + struct vfio_pci_core_device core_dev; + u8 config_space[PCI_CONFIG_SPACE_LENGTH]; + + void __iomem *bar0_map; + + u8 **vgpu_types; + NVA081_CTRL_VGPU_INFO *curr_vgpu_type; + u32 num_vgpu_types; + + struct nvidia_vgpu_mgr *vgpu_mgr; + struct nvidia_vgpu *vgpu; +}; + +void nvidia_vgpu_vfio_setup_config(struct nvidia_vgpu_vfio *nvdev); +ssize_t nvidia_vgpu_vfio_access(struct nvidia_vgpu_vfio *nvdev, + char __user *buf, size_t count, + loff_t ppos, bool iswrite); + +#endif /* _NVIDIA_VGPU_VFIO_H__ */ diff --git a/drivers/vfio/pci/nvidia-vgpu/vfio_access.c b/drivers/vfio/pci/nvidia-vgpu/vfio_access.c new file mode 100644 index 000000000000..320c72a07dbe --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vfio_access.c @@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include +#include +#include + +#include "vfio.h" + +void nvidia_vgpu_vfio_setup_config(struct nvidia_vgpu_vfio *nvdev) +{ + u8 *buffer = NULL; + + memset(nvdev->config_space, 0, sizeof(nvdev->config_space)); + + /* Header type 0 (normal devices) */ + *(u16 *)&nvdev->config_space[PCI_VENDOR_ID] = 0x10de; + *(u16 *)&nvdev->config_space[PCI_DEVICE_ID] = + FIELD_GET(GENMASK(31, 16), nvdev->curr_vgpu_type->vdevId); + *(u16 *)&nvdev->config_space[PCI_COMMAND] = 0x0000; + *(u16 *)&nvdev->config_space[PCI_STATUS] = 0x0010; + + buffer = &nvdev->config_space[PCI_CLASS_REVISION]; + pci_read_config_byte(nvdev->core_dev.pdev, PCI_CLASS_REVISION, buffer); + + nvdev->config_space[PCI_CLASS_PROG] = 0; /* VGA-compatible */ + nvdev->config_space[PCI_CLASS_DEVICE] = 0; /* VGA controller */ + nvdev->config_space[PCI_CLASS_DEVICE + 1] = 3; /* display controller */ + + /* BAR0: 32-bit */ + *(u32 *)&nvdev->config_space[PCI_BASE_ADDRESS_0] = 0x00000000; + /* BAR1: 64-bit, prefetchable */ + *(u32 *)&nvdev->config_space[PCI_BASE_ADDRESS_1] = 0x0000000c; + /* BAR2: 64-bit, prefetchable */ + *(u32 *)&nvdev->config_space[PCI_BASE_ADDRESS_3] = 0x0000000c; + /* Disable BAR3: I/O */ + *(u32 *)&nvdev->config_space[PCI_BASE_ADDRESS_5] = 0x00000000; + + *(u16 *)&nvdev->config_space[PCI_SUBSYSTEM_VENDOR_ID] = 0x10de; + *(u16 *)&nvdev->config_space[PCI_SUBSYSTEM_ID] = + FIELD_GET(GENMASK(15, 0), nvdev->curr_vgpu_type->vdevId); + + nvdev->config_space[PCI_CAPABILITY_LIST] = CAP_LIST_NEXT_PTR_MSIX; + nvdev->config_space[CAP_LIST_NEXT_PTR_MSIX + 1] = 0x0; + + /* INTx disabled */ + nvdev->config_space[0x3d] = 0; +} + +static void read_hw_pci_config(struct pci_dev *pdev, char *buf, + size_t count, loff_t offset) +{ + switch (count) { + case 4: + pci_read_config_dword(pdev, offset, (u32 *)buf); + break; + + case 2: + pci_read_config_word(pdev, offset, (u16 *)buf); + break; + + case 1: + pci_read_config_byte(pdev, offset, (u8 *)buf); + break; + default: + WARN_ONCE(1, "Not supported access len\n"); + break; + } +} + +static void write_hw_pci_config(struct pci_dev *pdev, char *buf, + size_t count, loff_t offset) +{ + switch (count) { + case 4: + pci_write_config_dword(pdev, offset, *(u32 *)buf); + break; + + case 2: + pci_write_config_word(pdev, offset, *(u16 *)buf); + break; + + case 1: + pci_write_config_byte(pdev, offset, *(u8 *)buf); + break; + default: + WARN_ONCE(1, "Not supported access len\n"); + break; + } +} + +static void hw_pci_config_rw(struct pci_dev *pdev, char *buf, + size_t count, loff_t offset, + bool is_write) +{ + is_write ? write_hw_pci_config(pdev, buf, count, offset) : + read_hw_pci_config(pdev, buf, count, offset); +} + +static ssize_t bar0_rw(struct nvidia_vgpu_vfio *nvdev, char *buf, + size_t count, loff_t ppos, bool iswrite) +{ + struct pci_dev *pdev = nvdev->core_dev.pdev; + int index = VFIO_PCI_OFFSET_TO_INDEX(ppos); + loff_t offset = ppos; + void __iomem *map; + u32 val; + int ret; + + if (index != VFIO_PCI_BAR0_REGION_INDEX) + return -EINVAL; + + offset &= VFIO_PCI_OFFSET_MASK; + + if (nvdev->bar0_map == NULL) { + ret = pci_request_selected_regions(pdev, 1 << index, "nvidia-vgpu-vfio"); + if (ret) + return ret; + + if (!(pci_resource_flags(pdev, index) & IORESOURCE_MEM)) { + pci_release_selected_regions(pdev, 1 << index); + return -EIO; + } + + map = ioremap(pci_resource_start(pdev, index), pci_resource_len(pdev, index)); + if (!map) { + pci_err(pdev, "Can't map BAR0 MMIO space\n"); + pci_release_selected_regions(pdev, 1 << index); + return -ENOMEM; + } + nvdev->bar0_map = map; + } else + map = nvdev->bar0_map; + + if (!iswrite) { + switch (count) { + case 4: + val = ioread32(map + offset); + break; + case 2: + val = ioread16(map + offset); + break; + case 1: + val = ioread8(map + offset); + break; + } + memcpy(buf, (u8 *)&val, count); + } else { + switch (count) { + case 4: + iowrite32(*(u32 *)buf, map + offset); + break; + case 2: + iowrite16(*(u16 *)buf, map + offset); + break; + case 1: + iowrite8(*(u8 *)buf, map + offset); + break; + } + } + return count; +} + +static ssize_t pci_config_rw(struct nvidia_vgpu_vfio *nvdev, char *buf, + size_t count, loff_t ppos, bool iswrite) +{ + struct pci_dev *pdev = nvdev->core_dev.pdev; + int index = VFIO_PCI_OFFSET_TO_INDEX(ppos); + loff_t offset = ppos; + u32 bar_mask, cfg_addr; + u32 val = 0; + + if (index != VFIO_PCI_CONFIG_REGION_INDEX) + return -EINVAL; + + offset &= VFIO_PCI_OFFSET_MASK; + + if ((offset >= CAP_LIST_NEXT_PTR_MSIX) && (offset < + (CAP_LIST_NEXT_PTR_MSIX + MSIX_CAP_SIZE))) { + hw_pci_config_rw(pdev, buf, count, offset, iswrite); + return count; + } + + if (!iswrite) { + memcpy(buf, (u8 *)&nvdev->config_space[offset], count); + + switch (offset) { + case PCI_COMMAND: + hw_pci_config_rw(pdev, (char *)&val, count, offset, iswrite); + + switch (count) { + case 4: + val = (u32)(val & 0xFFFF0000) | (val & + (PCI_COMMAND_PARITY | PCI_COMMAND_SERR)); + break; + case 2: + val = (val & (PCI_COMMAND_PARITY | PCI_COMMAND_SERR)); + break; + default: + WARN_ONCE(1, "Not supported access len\n"); + break; + } + break; + case PCI_STATUS: + hw_pci_config_rw(pdev, (char *)&val, count, offset, iswrite); + break; + + default: + break; + } + *(u32 *)buf = *(u32 *)buf | val; + } else { + switch (offset) { + case PCI_VENDOR_ID: + case PCI_DEVICE_ID: + case PCI_CAPABILITY_LIST: + break; + + case PCI_STATUS: + hw_pci_config_rw(pdev, buf, count, offset, iswrite); + break; + + case PCI_COMMAND: + if (count == 4) { + val = (u32)((*(u32 *)buf & 0xFFFF0000) >> 16); + hw_pci_config_rw(pdev, (char *)&val, 2, PCI_STATUS, iswrite); + + val = (u32)(*(u32 *)buf & 0x0000FFFF); + *(u32 *)buf = val; + } + + memcpy((u8 *)&nvdev->config_space[offset], buf, count); + break; + + case PCI_BASE_ADDRESS_0: + case PCI_BASE_ADDRESS_1: + case PCI_BASE_ADDRESS_2: + case PCI_BASE_ADDRESS_3: + case PCI_BASE_ADDRESS_4: + cfg_addr = *(u32 *)buf; + + switch (offset) { + case PCI_BASE_ADDRESS_0: + bar_mask = (u32)((~(pci_resource_len(pdev, VFIO_PCI_BAR0_REGION_INDEX)) + 1) & ~0xFul); + cfg_addr = (cfg_addr & bar_mask) | (nvdev->config_space[offset] & 0xFul); + break; + case PCI_BASE_ADDRESS_1: + bar_mask = (u32)((~(nvdev->curr_vgpu_type->bar1Length * 1024 * 1024) + 1) & ~0xFul); + cfg_addr = (cfg_addr & bar_mask) | (nvdev->config_space[offset] & 0xFul); + break; + + case PCI_BASE_ADDRESS_2: + bar_mask = (u32)(((~(nvdev->curr_vgpu_type->bar1Length * 1024 * 1024) + 1) & ~0xFul) >> 32); + cfg_addr = (cfg_addr & bar_mask); + break; + + case PCI_BASE_ADDRESS_3: + bar_mask = (u32)((~(pci_resource_len(pdev, VFIO_PCI_BAR3_REGION_INDEX)) + 1) & ~0xFul); + cfg_addr = (cfg_addr & bar_mask) | (nvdev->config_space[offset] & 0xFul); + break; + + case PCI_BASE_ADDRESS_4: + bar_mask = (u32)(((~(pci_resource_len(pdev, VFIO_PCI_BAR3_REGION_INDEX)) + 1) & ~0xFul) >> 32); + cfg_addr = (cfg_addr & bar_mask); + break; + } + *(u32 *)&nvdev->config_space[offset] = cfg_addr; + break; + default: + break; + + } + } + return count; +} + +ssize_t nvidia_vgpu_vfio_access(struct nvidia_vgpu_vfio *nvdev, char *buf, + size_t count, loff_t ppos, bool iswrite) +{ + int index = VFIO_PCI_OFFSET_TO_INDEX(ppos); + + if (index >= VFIO_PCI_NUM_REGIONS) + return -EINVAL; + + switch (index) { + case VFIO_PCI_CONFIG_REGION_INDEX: + return pci_config_rw(nvdev, buf, count, ppos, + iswrite); + case VFIO_PCI_BAR0_REGION_INDEX: + return bar0_rw(nvdev, buf, count, ppos, iswrite); + default: + return -EINVAL; + } + return count; +} diff --git a/drivers/vfio/pci/nvidia-vgpu/vfio_main.c b/drivers/vfio/pci/nvidia-vgpu/vfio_main.c new file mode 100644 index 000000000000..667ed6fb48f6 --- /dev/null +++ b/drivers/vfio/pci/nvidia-vgpu/vfio_main.c @@ -0,0 +1,511 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * Copyright © 2024 NVIDIA Corporation + */ + +#include +#include +#include +#include +#include +#include + +#include "vfio.h" + +static int pdev_to_gfid(struct pci_dev *pdev) +{ + return pci_iov_vf_id(pdev) + 1; +} + +static int destroy_vgpu(struct nvidia_vgpu_vfio *nvdev) +{ + int ret; + + ret = nvidia_vgpu_mgr_destroy_vgpu(nvdev->vgpu); + if (ret) + return ret; + + kfree(nvdev->vgpu); + nvdev->vgpu = NULL; + return 0; +} + +static int create_vgpu(struct nvidia_vgpu_vfio *nvdev) +{ + struct nvidia_vgpu_mgr *vgpu_mgr = nvdev->vgpu_mgr; + struct pci_dev *pdev = nvdev->core_dev.pdev; + struct nvidia_vgpu *vgpu; + int ret; + + vgpu = kzalloc(sizeof(*vgpu), GFP_KERNEL); + if (!vgpu) + return -ENOMEM; + + vgpu->info.id = pci_iov_vf_id(pdev); + vgpu->info.dbdf = (0 << 16) | pci_dev_id(pdev); + vgpu->info.gfid = pdev_to_gfid(pdev); + + vgpu->vgpu_mgr = vgpu_mgr; + vgpu->pdev = pdev; + + ret = nvidia_vgpu_mgr_create_vgpu(vgpu, + (u8 *)nvdev->curr_vgpu_type); + if (ret) { + kfree(vgpu); + return ret; + } + + pr_err("create_vgpu() called\n"); + nvdev->vgpu = vgpu; + return 0; +} + +static inline struct vfio_pci_core_device * +vdev_to_core_dev(struct vfio_device *vdev) +{ + return container_of(vdev, struct vfio_pci_core_device, vdev); +} + +static inline struct nvidia_vgpu_vfio * +core_dev_to_nvdev(struct vfio_pci_core_device *core_dev) +{ + return container_of(core_dev, struct nvidia_vgpu_vfio, core_dev); +} + +static void detach_vgpu_mgr(struct nvidia_vgpu_vfio *nvdev) +{ + nvidia_vgpu_mgr_put(nvdev->vgpu_mgr); + + nvdev->vgpu_mgr = NULL; + nvdev->vgpu_types = NULL; + nvdev->num_vgpu_types = 0; +} + +static int attach_vgpu_mgr(struct nvidia_vgpu_vfio *nvdev, + struct pci_dev *pdev) +{ + struct nvidia_vgpu_mgr *vgpu_mgr; + + vgpu_mgr = nvidia_vgpu_mgr_get(pdev); + if (IS_ERR(vgpu_mgr)) + return PTR_ERR(vgpu_mgr); + + nvdev->vgpu_mgr = vgpu_mgr; + nvdev->vgpu_types = nvdev->vgpu_mgr->vgpu_types; + nvdev->num_vgpu_types = nvdev->vgpu_mgr->num_vgpu_types; + + return 0; +} + +static NVA081_CTRL_VGPU_INFO * +find_vgpu_type(struct nvidia_vgpu_vfio *nvdev, u32 type_id) +{ + NVA081_CTRL_VGPU_INFO *vgpu_type; + u32 i; + + for (i = 0; i < nvdev->num_vgpu_types; i++) { + vgpu_type = (NVA081_CTRL_VGPU_INFO *)nvdev->vgpu_types[i]; + if (vgpu_type->vgpuType == type_id) + return vgpu_type; + } + + return NULL; +} + +static int +nvidia_vgpu_vfio_open_device(struct vfio_device *vdev) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + struct pci_dev *pdev = core_dev->pdev; + u64 pf_dma_mask; + int ret; + + if (!nvdev->curr_vgpu_type) + return -ENODEV; + + if (!pdev->physfn) + return -EINVAL; + + ret = create_vgpu(nvdev); + if (ret) + return ret; + + ret = pci_enable_device(pdev); + if (ret) + goto err_enable_device; + + pci_set_master(pdev); + + pf_dma_mask = dma_get_mask(&pdev->physfn->dev); + dma_set_mask(&pdev->dev, pf_dma_mask); + dma_set_coherent_mask(&pdev->dev, pf_dma_mask); + + ret = pci_try_reset_function(pdev); + if (ret) + goto err_reset_function; + + ret = nvidia_vgpu_mgr_enable_bme(nvdev->vgpu); + if (ret) + goto err_enable_bme; + + return 0; + +err_enable_bme: +err_reset_function: + pci_clear_master(pdev); + pci_disable_device(pdev); +err_enable_device: + destroy_vgpu(nvdev); + return ret; +} + +static void +nvidia_vgpu_vfio_close_device(struct vfio_device *vdev) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + struct pci_dev *pdev = core_dev->pdev; + + WARN_ON(destroy_vgpu(nvdev)); + + if (nvdev->bar0_map) { + iounmap(nvdev->bar0_map); + pci_release_selected_regions(pdev, 1 << 0); + nvdev->bar0_map = NULL; + } + + pci_clear_master(pdev); + pci_disable_device(pdev); +} + +static int +get_region_info(struct vfio_pci_core_device *core_dev, unsigned long arg) +{ + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + struct pci_dev *pdev = core_dev->pdev; + struct vfio_region_info info; + unsigned long minsz; + int ret = 0; + + minsz = offsetofend(struct vfio_region_info, offset); + if (copy_from_user(&info, (void __user *)arg, minsz)) + return -EINVAL; + + if (info.argsz < minsz) + return -EINVAL; + + switch (info.index) { + case VFIO_PCI_CONFIG_REGION_INDEX: + info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index); + info.size = PCI_CONFIG_SPACE_LENGTH; + info.flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE; + break; + + case VFIO_PCI_BAR0_REGION_INDEX ... VFIO_PCI_BAR4_REGION_INDEX: + struct vfio_info_cap caps = { .buf = NULL, .size = 0 }; + + info.offset = VFIO_PCI_INDEX_TO_OFFSET(info.index); + info.size = pci_resource_len(pdev, info.index); + + if (info.index == VFIO_PCI_BAR1_REGION_INDEX) + info.size = nvdev->curr_vgpu_type->bar1Length * 1024 * 1024; + + if (!info.size) { + info.flags = 0; + break; + } + info.flags = VFIO_REGION_INFO_FLAG_READ | + VFIO_REGION_INFO_FLAG_WRITE | + VFIO_REGION_INFO_FLAG_MMAP; + + if (caps.size) { + info.flags |= VFIO_REGION_INFO_FLAG_CAPS; + if (info.argsz < sizeof(info) + caps.size) { + info.argsz = sizeof(info) + caps.size; + info.cap_offset = 0; + } else { + vfio_info_cap_shift(&caps, sizeof(info)); + if (copy_to_user((void __user *)arg + + sizeof(info), caps.buf, + caps.size)) { + kfree(caps.buf); + ret = -EFAULT; + break; + } + info.cap_offset = sizeof(info); + } + kfree(caps.buf); + } + break; + case VFIO_PCI_BAR5_REGION_INDEX: + case VFIO_PCI_ROM_REGION_INDEX: + case VFIO_PCI_VGA_REGION_INDEX: + info.size = 0; + break; + + default: + if (info.index >= VFIO_PCI_NUM_REGIONS) + ret = -EINVAL; + break; + } + + if (!ret) + ret = copy_to_user((void __user *)arg, &info, minsz) ? -EFAULT : 0; + + return ret; +} + +static long nvidia_vgpu_vfio_ioctl(struct vfio_device *vdev, + unsigned int cmd, + unsigned long arg) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + int ret = 0; + + if (!nvdev->curr_vgpu_type) + return -ENODEV; + + switch (cmd) { + case VFIO_DEVICE_GET_REGION_INFO: + ret = get_region_info(core_dev, arg); + break; + case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO: + case VFIO_DEVICE_PCI_HOT_RESET: + case VFIO_DEVICE_RESET: + break; + + default: + ret = vfio_pci_core_ioctl(vdev, cmd, arg); + break; + } + + return ret; +} + +static ssize_t nvidia_vgpu_vfio_read(struct vfio_device *vdev, + char __user *buf, size_t count, + loff_t *ppos) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + u64 val; + size_t done = 0; + int ret = 0, size; + + if (!nvdev->curr_vgpu_type) + return -ENODEV; + + while (count) { + if (count >= 4 && !(*ppos % 4)) + size = 4; + else if (count >= 2 && !(*ppos % 2)) + size = 2; + else + size = 1; + + ret = nvidia_vgpu_vfio_access(nvdev, (char *)&val, size, *ppos, false); + + if (ret <= 0) + return ret; + + if (copy_to_user(buf, &val, size) != 0) + return -EFAULT; + + *ppos += size; + buf += size; + count -= size; + done += size; + } + + return done; +} + +static ssize_t nvidia_vgpu_vfio_write(struct vfio_device *vdev, + const char __user *buf, size_t count, + loff_t *ppos) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + u64 val; + size_t done = 0; + int ret = 0, size; + + if (!nvdev->curr_vgpu_type) + return -ENODEV; + + while (count) { + if (count >= 4 && !(*ppos % 4)) + size = 4; + else if (count >= 2 && !(*ppos % 2)) + size = 2; + else + size = 1; + + if (copy_from_user(&val, buf, size) != 0) + return -EFAULT; + + ret = nvidia_vgpu_vfio_access(nvdev, (char *)&val, size, *ppos, true); + + if (ret <= 0) + return ret; + + *ppos += size; + buf += size; + count -= size; + done += size; + } + + return done; +} + +static int nvidia_vgpu_vfio_mmap(struct vfio_device *vdev, + struct vm_area_struct *vma) +{ + struct vfio_pci_core_device *core_dev = vdev_to_core_dev(vdev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + struct pci_dev *pdev = core_dev->pdev; + u64 phys_len, req_len, pgoff, req_start; + unsigned int index; + + if (!nvdev->curr_vgpu_type) + return -ENODEV; + + index = vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT); + + if (index >= VFIO_PCI_BAR5_REGION_INDEX) + return -EINVAL; + if (vma->vm_end < vma->vm_start) + return -EINVAL; + if ((vma->vm_flags & VM_SHARED) == 0) + return -EINVAL; + + phys_len = PAGE_ALIGN(pci_resource_len(pdev, index)); + req_len = vma->vm_end - vma->vm_start; + pgoff = vma->vm_pgoff & + ((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + req_start = pgoff << PAGE_SHIFT; + + if (req_len == 0) + return -EINVAL; + + if ((req_start + req_len > phys_len) || (phys_len == 0)) + return -EINVAL; + + vma->vm_private_data = vdev; + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + vma->vm_pgoff = (pci_resource_start(pdev, index) >> PAGE_SHIFT) + pgoff; + vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); + + return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, req_len, vma->vm_page_prot); +} + +static const struct vfio_device_ops nvidia_vgpu_vfio_ops = { + .name = "nvidia-vgpu-vfio-pci", + .init = vfio_pci_core_init_dev, + .release = vfio_pci_core_release_dev, + .open_device = nvidia_vgpu_vfio_open_device, + .close_device = nvidia_vgpu_vfio_close_device, + .ioctl = nvidia_vgpu_vfio_ioctl, + .device_feature = vfio_pci_core_ioctl_feature, + .read = nvidia_vgpu_vfio_read, + .write = nvidia_vgpu_vfio_write, + .mmap = nvidia_vgpu_vfio_mmap, + .request = vfio_pci_core_request, + .match = vfio_pci_core_match, + .bind_iommufd = vfio_iommufd_physical_bind, + .unbind_iommufd = vfio_iommufd_physical_unbind, + .attach_ioas = vfio_iommufd_physical_attach_ioas, + .detach_ioas = vfio_iommufd_physical_detach_ioas, +}; + +static int setup_vgpu_type(struct nvidia_vgpu_vfio *nvdev) +{ + nvdev->curr_vgpu_type = find_vgpu_type(nvdev, 869); + if (!nvdev->curr_vgpu_type) + return -ENODEV; + return 0; +} + +static int nvidia_vgpu_vfio_probe(struct pci_dev *pdev, + const struct pci_device_id *id_table) +{ + struct nvidia_vgpu_vfio *nvdev; + int ret; + + if (!pdev->is_virtfn) + return -EINVAL; + + nvdev = vfio_alloc_device(nvidia_vgpu_vfio, core_dev.vdev, + &pdev->dev, &nvidia_vgpu_vfio_ops); + if (IS_ERR(nvdev)) + return PTR_ERR(nvdev); + + ret = attach_vgpu_mgr(nvdev, pdev); + if (ret) + goto err_attach_vgpu_mgr; + + ret = setup_vgpu_type(nvdev); + if (ret) + goto err_setup_vgpu_type; + + nvidia_vgpu_vfio_setup_config(nvdev); + + dev_set_drvdata(&pdev->dev, &nvdev->core_dev); + + ret = vfio_pci_core_register_device(&nvdev->core_dev); + if (ret) + goto err_setup_vgpu_type; + + return 0; + +err_setup_vgpu_type: + detach_vgpu_mgr(nvdev); + +err_attach_vgpu_mgr: + vfio_put_device(&nvdev->core_dev.vdev); + + pci_err(pdev, "VF probe failed with ret: %d\n", ret); + return ret; +} + +static void nvidia_vgpu_vfio_remove(struct pci_dev *pdev) +{ + struct vfio_pci_core_device *core_dev = dev_get_drvdata(&pdev->dev); + struct nvidia_vgpu_vfio *nvdev = core_dev_to_nvdev(core_dev); + + vfio_pci_core_unregister_device(core_dev); + detach_vgpu_mgr(nvdev); + vfio_put_device(&core_dev->vdev); +} + +struct pci_device_id nvidia_vgpu_vfio_table[] = { + { + .vendor = PCI_VENDOR_ID_NVIDIA, + .device = PCI_ANY_ID, + .subvendor = PCI_ANY_ID, + .subdevice = PCI_ANY_ID, + .class = (PCI_CLASS_DISPLAY_3D << 8), + .class_mask = ~0, + }, + { } +}; +MODULE_DEVICE_TABLE(pci, nvidia_vgpu_vfio_table); + +struct pci_driver nvidia_vgpu_vfio_driver = { + .name = "nvidia-vgpu-vfio", + .id_table = nvidia_vgpu_vfio_table, + .probe = nvidia_vgpu_vfio_probe, + .remove = nvidia_vgpu_vfio_remove, + .driver_managed_dma = true, +}; + +module_pci_driver(nvidia_vgpu_vfio_driver); + +MODULE_LICENSE("Dual MIT/GPL"); +MODULE_AUTHOR("Vinay Kabra "); +MODULE_AUTHOR("Kirti Wankhede "); +MODULE_AUTHOR("Zhi Wang "); +MODULE_DESCRIPTION("NVIDIA vGPU VFIO Variant Driver - User Level driver for NVIDIA vGPU"); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu.c b/drivers/vfio/pci/nvidia-vgpu/vgpu.c index 93d27db30a41..003ca116b4a8 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu.c +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu.c @@ -328,3 +328,25 @@ int nvidia_vgpu_mgr_create_vgpu(struct nvidia_vgpu *vgpu, u8 *vgpu_type) return ret; } EXPORT_SYMBOL(nvidia_vgpu_mgr_create_vgpu); + +static int update_bme_state(struct nvidia_vgpu *vgpu) +{ + NV_VGPU_CPU_RPC_DATA_UPDATE_BME_STATE params = {0}; + + params.enable = true; + + return nvidia_vgpu_rpc_call(vgpu, NV_VGPU_CPU_RPC_MSG_UPDATE_BME_STATE, + ¶ms, sizeof(params)); +} + +/** + * nvidia_vgpu_enable_bme - handle BME sequence + * @vf: the vGPU instance + * + * Returns: 0 on success, others on failure. + */ +int nvidia_vgpu_mgr_enable_bme(struct nvidia_vgpu *vgpu) +{ + return update_bme_state(vgpu); +} +EXPORT_SYMBOL(nvidia_vgpu_mgr_enable_bme); diff --git a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h index af922d8e539c..2c9e0eebcb99 100644 --- a/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h +++ b/drivers/vfio/pci/nvidia-vgpu/vgpu_mgr.h @@ -84,6 +84,6 @@ int nvidia_vgpu_rpc_call(struct nvidia_vgpu *vgpu, u32 msg_type, void nvidia_vgpu_clean_rpc(struct nvidia_vgpu *vgpu); int nvidia_vgpu_setup_rpc(struct nvidia_vgpu *vgpu); -int nvidia_vgpu_mgr_reset_vgpu(struct nvidia_vgpu *vgpu); +int nvidia_vgpu_mgr_enable_bme(struct nvidia_vgpu *vgpu); #endif