From patchwork Sun Sep 22 12:49:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhi Wang X-Patchwork-Id: 13809103 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2081.outbound.protection.outlook.com [40.107.95.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 812931803A for ; Sun, 22 Sep 2024 12:50:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009450; cv=fail; b=f0ZJQkJaySNwlRoTlJf0TqKlbmJvbVS4kWqIIjWPiIsfXQ74bB9NmlGPZmWd9NujHcn5+TQqXqpzjBQLSpR2Xk9G3aatdIH222CmynpvA+Xh0GBzEPclg8Pgw9xKnoreb9AF7k4lC8l3yYFfmSA9ofcsirJBz6CFkUoktql7E1k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727009450; c=relaxed/simple; bh=mvaErpb73mjTNY/zd1F0CvK3v2ZX4b/VVqTY/1RnjF8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Q2IgYkhbW+yAOXLZPpq5Pymd5IlLO+8B26zEY80kv+rd/M67IhOfyJOvOZxSsNGs+57kvFcKW25wRVT8NxFxZk5Ubv5t5e+/9h1VS6YuYcVSTCUXdfZBo4GY9XfpsSP6fhRBzJECgtIu6uSM810dhQ1SkcOcziwumLTPzl2vjUY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=DI61n69/; arc=fail smtp.client-ip=40.107.95.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="DI61n69/" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EJtJ3oQwuCXyFZ90UWpUXxN+Q46L4nsU7egHncUSr+DVXrMMOaB3foR1rWJrQLBoU8jBLcVleQ9/Lu6/kHcFLwBOswajZTCFcdAD9O4LeEU1YQqvDkRdI2nPNIi/piaNaFBTLpoZbL1Ijqu0ZyAzpLopcJDuU9oBsOAXSM1ZpUQHfr/ehL8e+rC2GRHHg+fDaSixvwOcgyiKzZmD8+f7zizQLjVtsAi/S1wK8Rfhzd0XtM7oQ7uUrCEvGGPws3dfAjSDpbeSVQu12moD+eTB/xCTKpgQHgTAKJrs9mYj3QrP/j/bYSkmYTL0oTHJJ/WWA3vm+VnI0i5bvuGrsaCtyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NYE+Aay3FUrAyLLloByMaZmI4Qgq2DLngBI4ghqmorM=; b=H88FGj0oZiWuCfMhIayNWG5nDxYq+6TkQQXAW3WuweVwpHi0S3NVY9+nr18joo4DwotdosSGJmrDJ0gUGBcjGAom8zaBCHyDmWd9lBJy+Sk/elh9aieWnL0fmdLsIgS+MKkRkGYTHPmmjvYwd2FKNiNotf5xvWCRk5zi02OcwOW07gpTso1QeqZuSLqAVfBHgVsRGcTQti9xlp4Pl6hmeiqLCGmDooCHQUpgJgLQaLRtWdluti4wP7zE0k/p5OQlUVFMM6HVJ5BsPQMfL/K7HhuVqBtbV5SMM0GEawDuL34LmbFKsQ/lucIgpXCSOCY1Pjg8ElxtwKbYpxpFvnarQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NYE+Aay3FUrAyLLloByMaZmI4Qgq2DLngBI4ghqmorM=; b=DI61n69/aCIqCw+K9H8QaObnZItW5wSTWWlIquV1czxXgAS0ADswyLBjt1wZQS2OrqWwDC3dQNpySkT0Y9o5MiurpdeLBuhYgR7EFhn/jBi9OWCCThS1lkh14/ytmcTCkLhiT8k2cODPkuUMWPApSDrkiItW+t8VBU2Ryx3WYoby7gIUGDNVOpU43OyyC82AzVqatvGtSV+olX3BuEnFjGX5pWV1nJw8TAKMqjooJ2nuRr0VbETneIJz+ZTX3SvtQL/CEuI4RfBXOreGxV02WdDm969GE7wvIdccEUfpEWZuEPqmt4qknYbfv9+ibXYZXVbR+EzUCFmwN/qbQqIw+A== Received: from BN8PR04CA0055.namprd04.prod.outlook.com (2603:10b6:408:d4::29) by DM6PR12MB4219.namprd12.prod.outlook.com (2603:10b6:5:217::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25; Sun, 22 Sep 2024 12:50:44 +0000 Received: from BL6PEPF00020E5F.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::f0) by BN8PR04CA0055.outlook.office365.com (2603:10b6:408:d4::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.30 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF00020E5F.mail.protection.outlook.com (10.167.249.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Sun, 22 Sep 2024 12:50:44 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 22 Sep 2024 05:50:29 -0700 Received: from inno-linux.nvidia.com (10.127.8.13) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 22 Sep 2024 05:50:29 -0700 From: Zhi Wang To: , CC: , , , , , , , , , , , , , Subject: [RFC 15/29] nvkm/vgpu: introduce FB memory allocation for vGPU Date: Sun, 22 Sep 2024 05:49:37 -0700 Message-ID: <20240922124951.1946072-16-zhiw@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240922124951.1946072-1-zhiw@nvidia.com> References: <20240922124951.1946072-1-zhiw@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF00020E5F:EE_|DM6PR12MB4219:EE_ X-MS-Office365-Filtering-Correlation-Id: ff7c44cf-83fd-42c8-e533-08dcdb052c3a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|82310400026; X-Microsoft-Antispam-Message-Info: WNrmRG1cBUBub4DPI1yKf8osMEQrxqxEmLezW0L/X1JThSyI2gsRMvtfiCsxrIBYeMdd655BY8h63BMONU0tBxx3quoURE3754YbNWnTDnDXeFLD+AuRYbulXQ4djTzov2XOaUq6djOXNfaCke2Rxtaq4T+Nwlm6xteV1WjtLioADR4Tk1fjvIL28v2y91G1pKJzJpXyG5GFCsGxF1bQ0aayi396NpNk0oJ37ukbAtm7H0uQ5ftSAMW/J3LlJIsmrt1Ngg6eGXrS63d6U1ZpVCOndPrlVhZUDdZ9kW5c2fly++BxawFCVJbOFiqRc9XHdJvKPWwg2OP7pvy9DVVgy6qlxhidUe7JTAyvMcrUVMO8srwUfcAp/+xJV5ODM8Xb8rXHkBAP8k38RvI3h7ToV5+nL+uASjgzKugy1jMR5ucMiTkBn9lUH7SnE9K08BazoIEZsoMAfnOD8RbRPUy46wedrIA0aSJozrGQjLP1XGVGuaDdDzK3vQOVTDxOuuIdZ6jaXJOSF2D1DcSlRy1d26aCsCx/y4UY5OPkV11hNYq5el9lDiKxtER1fvkBzXFJ626/a2FXx5ufl5nxVHAcTqWHDU0MHZT5nTImaLVsbQBfEh98Vj59saORqNXvobq4oT8xXTLpPhUvwJsGYTUaW24AWfEC3zFLx3tkGwCPB0t8M+mEXaiHq5kqKsN37fZT0nEGesbOW9RBl8TyST16TVqZLAl76gqE+BpTTXyfLyeSo40Ejpp6reGcvrwGuAaWoYnC66V/sojN0zlwoPZ3mL02Y2rWyj9yrvADCo1wZTheA12wfSk98cRW6zcCDtNOsi1tjH7g67EHcfVg8N8k81VldBdw/ATydTzuyVi97M2euxks9t3PFPyrIhR8hmh317J9viWNZY9uCptPlZHI/HiS+7o/aBdOpEg9msJrbBRMlgCaktTpNL6W6djQjd69jqwAi9JpdsEJnGH4sNuD1TluQ5VDgqHP9KOkpDatcSzqX/vpq/ZvLHmTcz2BjJ0TIJ4pJSB+aduLCgODRJ4uVas4fKCkgz8tlKT5RfiB9vJ9zFsFh38LQxQ9PTUyy8Dr9PT8WYXYcnsmJEmoAgDWtuhLNkiMe7wf70DmJGiHNe3XTWGlr7yMojK/uWzxgga/gXKSTnsvQTStsuoFeXQTA56J+cwrtTto8kwDqCi3ZgpPmDjg0IbNQk8d8B2wIfL3oBuKTZsj9lwEQiyKBgOeLLghyDRH/zaJSeytHaYuJDWSpNUdqSqQy5oS1leI8WjPRBbEHu+sMAZMZzkVma3feiNCJ//HXRIfLZrf5u654CvmZ7FoidGuW+B/RxcPw3ynjvFXH7/dCu5rLyw24Y/ABjWW2cVOm/ruDHhfrqHI128aBBPEp2nqeETs0+gSb/XJvXecvTIe5TrWYkgbocruELZUeZCnHrYyAdiYElzmAXi4mHUwoR8Niji8GCLzZsdb X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2024 12:50:44.1113 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff7c44cf-83fd-42c8-e533-08dcdb052c3a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF00020E5F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4219 Creating a vGPU requires allocating a mgmt heap from the FB memory. The size of the mgmt heap that a vGPU requires is from the vGPU type. Expose the FB memory allocation to NVIDIA vGPU VFIO module to allocate the mgmt heap when creating a vGPU. Signed-off-by: Zhi Wang --- .../nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h | 6 +++ drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c | 51 +++++++++++++++++++ include/drm/nvkm_vgpu_mgr_vfio.h | 8 +++ 3 files changed, 65 insertions(+) diff --git a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h index a351e8bfc772..b6e0321a53ad 100644 --- a/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h +++ b/drivers/gpu/drm/nouveau/include/nvkm/vgpu_mgr/vgpu_mgr.h @@ -6,6 +6,12 @@ #define NVIDIA_MAX_VGPUS 2 +struct nvkm_vgpu_mem { + struct nvidia_vgpu_mem base; + struct nvkm_memory *mem; + struct nvkm_vgpu_mgr *vgpu_mgr; +}; + struct nvkm_vgpu_mgr { bool enabled; struct nvkm_device *nvkm_dev; diff --git a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c index 44d901a0474d..2aabb2c5f142 100644 --- a/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c +++ b/drivers/gpu/drm/nouveau/nvkm/vgpu_mgr/vfio.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -154,6 +155,54 @@ static int alloc_chids(void *handle, int count) return ret; } +static void free_fbmem(struct nvidia_vgpu_mem *base) +{ + struct nvkm_vgpu_mem *mem = + container_of(base, struct nvkm_vgpu_mem, base); + struct nvkm_vgpu_mgr *vgpu_mgr = mem->vgpu_mgr; + struct nvkm_device *device = vgpu_mgr->nvkm_dev; + + nvdev_debug(device, "free fb mem: addr %llx size %llx\n", + base->addr, base->size); + + nvkm_memory_unref(&mem->mem); + kfree(mem); +} + +static struct nvidia_vgpu_mem *alloc_fbmem(void *handle, u64 size, + bool vmmu_aligned) +{ + struct nvkm_device *device = handle; + struct nvkm_vgpu_mgr *vgpu_mgr = &device->vgpu_mgr; + struct nvidia_vgpu_mem *base; + struct nvkm_vgpu_mem *mem; + u32 shift = vmmu_aligned ? ilog2(vgpu_mgr->vmmu_segment_size) : + NVKM_RAM_MM_SHIFT; + int ret; + + mem = kzalloc(sizeof(*mem), GFP_KERNEL); + if (!mem) + return ERR_PTR(-ENOMEM); + + base = &mem->base; + + ret = nvkm_ram_get(device, NVKM_RAM_MM_NORMAL, 0x1, shift, size, + true, true, &mem->mem); + if (ret) { + kfree(mem); + return ERR_PTR(ret); + } + + mem->vgpu_mgr = vgpu_mgr; + base->addr = mem->mem->func->addr(mem->mem); + base->size = mem->mem->func->size(mem->mem); + + nvdev_debug(device, "alloc fb mem: addr %llx size %llx\n", + base->addr, base->size); + + return base; +} + struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .vgpu_mgr_is_enabled = vgpu_mgr_is_enabled, .get_handle = get_handle, @@ -168,6 +217,8 @@ struct nvkm_vgpu_mgr_vfio_ops nvkm_vgpu_mgr_vfio_ops = { .rm_ctrl_done = rm_ctrl_done, .alloc_chids = alloc_chids, .free_chids = free_chids, + .alloc_fbmem = alloc_fbmem, + .free_fbmem = free_fbmem, }; /** diff --git a/include/drm/nvkm_vgpu_mgr_vfio.h b/include/drm/nvkm_vgpu_mgr_vfio.h index 001306fb0b5b..4841e9cf0d40 100644 --- a/include/drm/nvkm_vgpu_mgr_vfio.h +++ b/include/drm/nvkm_vgpu_mgr_vfio.h @@ -16,6 +16,11 @@ struct nvidia_vgpu_gsp_client { void *gsp_device; }; +struct nvidia_vgpu_mem { + u64 addr; + u64 size; +}; + struct nvkm_vgpu_mgr_vfio_ops { bool (*vgpu_mgr_is_enabled)(void *handle); void (*get_handle)(void *handle, @@ -37,6 +42,9 @@ struct nvkm_vgpu_mgr_vfio_ops { void *ctrl); int (*alloc_chids)(void *handle, int count); void (*free_chids)(void *handle, int offset, int count); + struct nvidia_vgpu_mem *(*alloc_fbmem)(void *handle, u64 size, + bool vmmu_aligned); + void (*free_fbmem)(struct nvidia_vgpu_mem *mem); }; struct nvkm_vgpu_mgr_vfio_ops *nvkm_vgpu_mgr_get_vfio_ops(void *handle);