From patchwork Fri Jan 24 18:30:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13949823 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2089.outbound.protection.outlook.com [40.107.212.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5210821ADD4; Fri, 24 Jan 2025 18:31:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.89 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743498; cv=fail; b=okq1rz25P/6z4Wx9eH2pErRH0vWZTqUgYHKQyfT8FGJb5q/S8rq8kQfDRZf7SanbfDR+wTIULoscKJERQ84ZlRnsoFQDWXGIWZiGf/2I9do0LG5Zm0NVJQ6ZZFDNyvGyXE3WMzU/ztShHB+5RQs6GDZOsjN0f5RbOzJT45mIGpc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743498; c=relaxed/simple; bh=25frWz6E9qzJrdUA7y/bA33HODC1GGhpzua8cX5x4mQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qZa6UdpUbXzgHuMgivJawGqgPQLdjAIJubqt0P8NqNR0tNcNyP1vG/hHg9AJT+2MlZv/sVA/3G9Rr86zYz6c06kISbjIiNgwiFXUXxFxmVIUngW0DewKgOizxDjbVjGFu+FatMYT68ekGzwdjs5nN1ZdsLINCJEQq003jfg5p20= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=MeA7/zSg; arc=fail smtp.client-ip=40.107.212.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="MeA7/zSg" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RStFHNxp6B0W4pEkqg/Wd4t1zK7z/qKhIaHjTVuhtWmviBU9kXwVZUzDY3KXWB+/W8hClpuumJiK9pqGUL8iZqE7KPtJD5OPlHEhUitbWQpeVK2hDrfPbbG9VoFC9qoypIYkqjsf23VbxQpqCp+OOv9pdtsZSvVqHipytf1IwyNMEIi6DUytlr7qtLrT8zYvP84QnQri4Y8c6ApGiNOjwyYfRLcgTSk5+9kYfVCz2W9X5Czhf6Ns/Agqd6HNlygTvNxeUoz3pm6TjZohDPCG+JgOLvHbTIBN/xPCf1n2yj7YVA4BiKjEQbeUJR+TiZsBDTBTfmQ71EX9wC2jvEQlEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VtniKdu3qv2osqCpUpwQFA8+cObXJHXS3+82q8Z7Nz0=; b=WODxw8wXAmqKs4v/tS0WGW7dpKEbWaeC9gDuyAqGTiucJlnSNwaLTNJ2mneza+xzDjRhuq181zE+nuMtsyWRw4oe9OdTD398nwMy4WGl/lQ3iZ+rRB4MYog0u1s7lIpgMbCVdttEh55W+5NEAhjmgszV4XaRpt6C3tV/OvhEXi3HTcZwY46SCwI8dE29swDN/+r3Pro5DH2CFrE6470QoTqEeVomrWiJKk4j23h2p4UNYdtIwS78U8u1w/6P4SjcjG/HO1Mjy0BKS+9JyCs5mdAZtnn0c5djml8NRATgmCDkkPfxMTqAAPtOPrnoCuzBoqscbRPWoS8K83XQJlyOpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VtniKdu3qv2osqCpUpwQFA8+cObXJHXS3+82q8Z7Nz0=; b=MeA7/zSg0WH3qgKhPKv0XaYmqtbo9GvMqWjBhYdSlNhXaVDFVPCzmV7fFKX9uX1sxriszzN4cwW/z5Nsw4lWXru+hvuzYibMhb8GvqfsGszRzBvpiVuk11L2HLwouoFh5o4nir8NeuQkoQ6LLfg67O8oB3W4oXE5X45yDZUSx0uoqbW45wbmN4f58OHway9nA0503K+t7vMLhtcXhe6ayx35XnRaqNkDaWkFv1oQ7ontJb/RcV4U0hVq3aqfoL3DORPyR2HNiJxQRqM3lfxIWCyWEi79UmiXmTJ4EVkuMogfd39noo/PJ3jJsn4x3ZZMdoX1jmhBBn41tmBZPN/QXg== Received: from SA0PR12CA0003.namprd12.prod.outlook.com (2603:10b6:806:6f::8) by CH3PR12MB7691.namprd12.prod.outlook.com (2603:10b6:610:151::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.17; Fri, 24 Jan 2025 18:31:32 +0000 Received: from SN1PEPF00026369.namprd02.prod.outlook.com (2603:10b6:806:6f:cafe::74) by SA0PR12CA0003.outlook.office365.com (2603:10b6:806:6f::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8377.20 via Frontend Transport; Fri, 24 Jan 2025 18:31:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF00026369.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.8 via Frontend Transport; Fri, 24 Jan 2025 18:31:31 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 24 Jan 2025 10:31:15 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Fri, 24 Jan 2025 10:31:14 -0800 Received: from localhost.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14 via Frontend Transport; Fri, 24 Jan 2025 10:31:14 -0800 From: To: , , , , , , CC: , , , , , , , , , , , , , , , , Subject: [PATCH v6 1/4] vfio/nvgrace-gpu: Read dvsec register to determine need for uncached resmem Date: Fri, 24 Jan 2025 18:30:59 +0000 Message-ID: <20250124183102.3976-2-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250124183102.3976-1-ankita@nvidia.com> References: <20250124183102.3976-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026369:EE_|CH3PR12MB7691:EE_ X-MS-Office365-Filtering-Correlation-Id: 59de7d8c-b333-4d03-9a55-08dd3ca5531a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|36860700013|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: bebRb/YNa0C9gxxAIfSM/fL6sY9h6ec9e2bgh1MnaTGVbjJH0w2WlJx9ODJCcsZnoQS5MsUBTg4JCUKAXUxHcoT4QDswOZzHLCI9ndkUdTLdzO7Ershomx7m1fOZxbjtv0aRaqv91izZDxmMnxyhl1Iwqmb2/gHOt2dg96OdE3EdceTLN4ZCn7Ign/9SyqY96fm9sOTBrDBIg+etJXwVvaxAQKL975J3o4TcP+gO9j+aPPjxpGJqJpAHBUa66GK7RHzshieihsVUNCQWvG3AlYtq7J1ubvZA9JVTVl23pqex9zmR2TcOdDam0a2XVsmxqsnTpJoBV7fbgmDtjgetaGiCKUnQjz8uh0CIsg6ROLO7Et94gGpZ5X6ZkDlS/jAl1vnjhjiNXTqicbALvFke2e5N5CqzdU2lfOlYe9xz7LHsVLup8fRZU7FAlrxuNGleuIvYkTeGPxkgDypE69qEGq3JrvhRr9JhK05alDuOlUbO0sacDBPPLDoLIgFYTwOoCI5PB3LvTxW+SMJ2y4bxSr+0jLChfbeCzvlbOZ3GHKJ+DkjsvBy15bLwSwXzCEoYLyR3JE9vonz4QgswS2ar+Nrgiu9eb08bai0CIZew0tNavn83Np6sOnbXcQ2cqWSpgoxdQYXz9LMxMzmBXKH/BMY9foV5FA+h/kBvpHveBsAhrfFshnXOPA1jxXUHfTPO2wZyqIW1HPdxObNXt2XHbqf4bC485bFmfrArNI7tB0ol0lI6hKWGd36h7lq5l3eI/Mdo7skTVKp1zNv24m5ICBYudErQYwl+jFgv3KtbJnDJfAYhr7I4G93YcUjUtexWZsdMGzgnCC0onHidrxC7rrQWx2KY1LzLKhTVB3wFliW7l/7lFH+7hT+L4iSB+NNOlzpHE9LHhNsMGXvoiLVzr81K6WoHPuUIlxC+L+7Fh97G8XIuxdRboG9QOEzE4rFc+9WXKaQsnhTaHtom/2YtWBPFOcAs+ux0bMMa6+l6DYIXI8RTV3e4ioysevUBiOWtOAfLNi05midzgTMNM9iJ8+JgPs7Amqj5Mstvyq+jT9zQ6W+nz3lLPSyMqoKc3+gbD1G0ShZ1+qi7rY9NTxEEb4wlPHy49Fg6DEt8e9mkC28wAhXyPEspcLZty/CoaXtavTjjFTb44m5yd3003+MIsz87PHF/Ny/aYOEZs7lWkQSsuw2ARRk6Iaz9h11Y8iDZDXl8mYLB+wv+CeHBsYMM6HFRaU617LL5q1/N0LovpHcbv6u9/j5gK8mQujsGwE6cZRkctQIMq4npbh5N4mzm+l1el4sKWE2zqlZ3vkdp4a5zcWHnZV49EGiz/c2Z4NHdPWRQTZZ4AtOnGG+jcpgii+VvgeGoWNNZXwyV7H++Z3qYbitjmagMwvUZkNMSWTjG2AJxeu4ebH/NQkLuQrE63scGST3iGTIVyp+8/roq9l9Vw2qO2WTe3+s1AOqvUsLOkontd4f0IOIhe0s1XHdtgmY/am9ofg1ry5hu6NZNBoc= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(82310400026)(36860700013)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2025 18:31:31.7219 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 59de7d8c-b333-4d03-9a55-08dd3ca5531a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026369.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7691 From: Ankit Agrawal NVIDIA's recently introduced Grace Blackwell (GB) Superchip is a continuation with the Grace Hopper (GH) superchip that provides a cache coherent access to CPU and GPU to each other's memory with an internal proprietary chip-to-chip cache coherent interconnect. There is a HW defect on GH systems to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G region with uncached mapping carved out from the device memory. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue. This is fixed on the GB systems. The presence of the fix for the HW defect is communicated by the device firmware through the DVSEC PCI config register with ID 3. The module reads this to take a different codepath on GB vs GH. Scan through the DVSEC registers to identify the correct one and use it to determine the presence of the fix. Save the value in the device's nvgrace_gpu_pci_core_device structure. Link: https://www.nvidia.com/en-in/technologies/multi-instance-gpu/ [1] CC: Jason Gunthorpe CC: Kevin Tian Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/main.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index a467085038f0..b76368958d1c 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -23,6 +23,11 @@ /* A hardwired and constant ABI value between the GPU FW and VFIO driver. */ #define MEMBLK_SIZE SZ_512M +#define DVSEC_BITMAP_OFFSET 0xA +#define MIG_SUPPORTED_WITH_CACHED_RESMEM BIT(0) + +#define GPU_CAP_DVSEC_REGISTER 3 + /* * The state of the two device memory region - resmem and usemem - is * saved as struct mem_region. @@ -46,6 +51,7 @@ struct nvgrace_gpu_pci_core_device { struct mem_region resmem; /* Lock to control device memory kernel mapping */ struct mutex remap_lock; + bool has_mig_hw_bug; }; static void nvgrace_gpu_init_fake_bar_emu_regs(struct vfio_device *core_vdev) @@ -812,6 +818,26 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev, return ret; } +static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) +{ + int pcie_dvsec; + u16 dvsec_ctrl16; + + pcie_dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_NVIDIA, + GPU_CAP_DVSEC_REGISTER); + + if (pcie_dvsec) { + pci_read_config_word(pdev, + pcie_dvsec + DVSEC_BITMAP_OFFSET, + &dvsec_ctrl16); + + if (dvsec_ctrl16 & MIG_SUPPORTED_WITH_CACHED_RESMEM) + return false; + } + + return true; +} + static int nvgrace_gpu_probe(struct pci_dev *pdev, const struct pci_device_id *id) { @@ -832,6 +858,8 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, dev_set_drvdata(&pdev->dev, &nvdev->core_device); if (ops == &nvgrace_gpu_pci_ops) { + nvdev->has_mig_hw_bug = nvgrace_gpu_has_mig_hw_bug(pdev); + /* * Device memory properties are identified in the host ACPI * table. Set the nvgrace_gpu_pci_core_device structure. From patchwork Fri Jan 24 18:31:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13949826 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2051.outbound.protection.outlook.com [40.107.223.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F13C223B0; Fri, 24 Jan 2025 18:31:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743499; cv=fail; b=W2vEkB8MwA3vhQQVx1NNEq0dPHOLsr+kGn8cWo8snLMX+Qsn/8pnxfpJucDj1FSz5Q0HMRjkF1uKfF98PJHc5pGR6N+btBziGAaY5ErGZ7eUdgJyFcuJb5AlL/ZZK0+wU5yuzKXs/O1Co33GktuFwpnAQVww1BTZhEXHW6ji1Lo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743499; c=relaxed/simple; bh=mVDhl8gc8G/p75pwk3uy78Ni3VAWxdg7CSj8F5h7y5Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=izLV+4pnPjQ0fcM6/yihC021hDXg0OY1kMt9obs+hrrisBR5f1tMlI0GfGMX9UD+dqvxSC5CLuE828L7WRGrGlQFc3uEy9LThsTi513o678WCEdtDX6Oa9raYlHNJH7c8w9ffKmWa8XZJGqSfhRYakHItBYdpnafmfiEjpobo6A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=a859ZExG; arc=fail smtp.client-ip=40.107.223.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="a859ZExG" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hyWpZ1YrgqXE/craAYkKDevF1e9T1spThPRLE/+uNlL9x8dWmGgVDmWpUn9OLzq0bmx1L0oA6YkjSo+kBlH/uJOu/lh9AG1WUdL4/V2wbycNFdEk4+U6y7Gof0MPXHthliuBKaTTEmHVBBaV5t5saQn91+7epeNtKccyMVIR7nbZ0rggyj6FXBX7PaAsjXfZsFlCpxVysTdtQWKcQaty1Sgx2tJr5+7PQG8MzFZ2SPD5QVU0d8fQXl7biCaszJ0zK54SSat1j/trHltdwIfYO3owmB6Eg40fjNdl6ov+E9n63aUwu0VVw59mhFZgBnJrcHVwhri020zWSeEiYPPPug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hG2nK1PYoccYYsNyeHvpOdr1J8TvGa6Kd981LFx3fU0=; b=IerDI357KAulBfPb4ZChOxy2FwRwsGhgC/B2THPeAogzTr0PnEY9hMzk3phHPfp51XqQ2xFRySxV7+fkyKUpTIpu/icqqdvNTGNH4RRUsgUZpmE53DP4nm7FPLQxmo/Y0JoY6qUtIR7UuOIp4oYN7jgs3S8jXbmMmOZuwdu9dXDMCG5JPMpXoPHHFVDiQoXreWKSUpDr7PIg4efSSg0xA2gclcQubulzOd93t08FqLbkTDtsASYMlz4qACfYAauBMppXTkkEh/SWQDnG6TqUynu2ObK9Xi9s5vokGfmDUUqdPJQu5JahPSxWx+uI/wCzmVl+4JEcmNBIgEys6DclIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hG2nK1PYoccYYsNyeHvpOdr1J8TvGa6Kd981LFx3fU0=; b=a859ZExGTkuKU9E0PGykTuZXQLQzQU81tHaO89lZA3xwn6d7Bdeqyt1OkCsw0H8VXdetoO6uiHr4oORuhUR8Eceq7qGNaDWS3R4KgK8AoAxT6j8RvnQ/SYq40HEqvNX/7Ua6ELUyA7rWRa8M02dqB47IdacTVzZ/emUGGzqI6H+vVfaPPasH+LQ9RrQVNhZ1TkmhQ0IU/TKmeCz4jmXb5STKSjv/Udt7CeTZ12H49s8xoHQXuYQJRzZrXXJ6IJPYO9Zn0Rdu0l0MLG2aH+9HXrKAf/PysU/5RsZHG7AMZRsBEe08I+OiWA2tN9037P38soAloBSWUd0UQUQlAhHTPg== Received: from SA0PR12CA0011.namprd12.prod.outlook.com (2603:10b6:806:6f::16) by IA1PR12MB6602.namprd12.prod.outlook.com (2603:10b6:208:3a2::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8356.21; Fri, 24 Jan 2025 18:31:32 +0000 Received: from SN1PEPF00026369.namprd02.prod.outlook.com (2603:10b6:806:6f:cafe::2e) by SA0PR12CA0011.outlook.office365.com (2603:10b6:806:6f::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8377.19 via Frontend Transport; Fri, 24 Jan 2025 18:31:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF00026369.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.8 via Frontend Transport; Fri, 24 Jan 2025 18:31:32 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 24 Jan 2025 10:31:15 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Fri, 24 Jan 2025 10:31:15 -0800 Received: from localhost.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14 via Frontend Transport; Fri, 24 Jan 2025 10:31:15 -0800 From: To: , , , , , , CC: , , , , , , , , , , , , , , , , Subject: [PATCH v6 2/4] vfio/nvgrace-gpu: Expose the blackwell device PF BAR1 to the VM Date: Fri, 24 Jan 2025 18:31:00 +0000 Message-ID: <20250124183102.3976-3-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250124183102.3976-1-ankita@nvidia.com> References: <20250124183102.3976-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026369:EE_|IA1PR12MB6602:EE_ X-MS-Office365-Filtering-Correlation-Id: 9f5934af-697b-4f2a-bd5f-08dd3ca5537e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014|13003099007|7053199007; X-Microsoft-Antispam-Message-Info: QSPxtzz+1wvXm5jmfmvZgKQVipfO3BB9JrR9bqJ8AilGtL7Tu4DssmIs2yio8AzywcjvJc7tTTIP7RNdYt0UrCwR3xyKb6OQ11tOaEAyzpsGlBpXBfh47JIgK+9+OeN1WgIM3S4frrHwE4B37jIbBIZIY0ym1g53Ihe7QwwqjdEBbD5RyMcyw7ZbOkCNgUuR3TfaOrSLGUta0yJd+3sjXeALGcYrGA597jckyN1jXA9FhhYuvX3Go3pAFlwlI9gVTiXsWIQnofeasYE2MNEzE2BQx+qAdYeZeRATprJdGxsfBzncYRjnVzZgAS5DoTcDpy1daYVfItd2ErQ28y1zDXOB51o46Tc73MPJ3+K8xpqbU7Pvl66irWiV6s39bzd/G4Ej5UT9TmP8ywvywnYLeJliTvVwmhOwwJcBIWow2BlKnM+hIir/BqdBFJo17/kGsHGa6sPlco6XR8hsc5E62O+VhPRHm0r56Zf5h6Icfo/8dPFCOrEh71aepglSM6N0XjXUyFdbcMhCNKB5e0ItJy2ef7yVpWTnjTNf66/gq8MGOFGJf9EjCqiP0rpMkEud52I3kBp9CNveA/sXCJMU+6Qwq0M/wZgWKJx/1I5gWh/lnwNWistE8TAUnE4HJyEPgWETMmb3mzO8RwUq7nn/1w7oKnM2+QhbAs6UKyXEbwk2JdykMFLRkt9wg5kQ04b+ztUyjL43BZ7pWZMoSLoVoLZhD5vLHSv08POS1tfrSf22dIt7QWZvzNQa37kkrkYQWSx/h8MBIAX0XKHZT+8NiD1XAsEhwa6rNM4m41wH3yxM/pmqas6rbynlCbX6Fx9aWpyPjEgaqCOa83u0kNkDT1EuX4VD6+edQehpwdpt12zxkwvwLG0qIqi9P4pGujRSB4Xt95t9eYKw3+oPnxNAFYVPunWyMICHVH5wGew1Krkobl9zysKMVzx5zrFjyC6oSWRTF+g+mrNMCu71FUTVwD6lB1GL5pfpyciUojd7Jbbl5LIhQOkkJ1NA0bOLJ0xD7uihT/KR8Hn61SLcv09FHY3lWq7p/TTFVvjT2gktCQGbdaybfNjsnKXsHzNGiiujBMcfuVikKfgpiLW/GYzXSQoXWXN95jakxJ441bBHVTfbeGWsRhusjN1MOj2cU4/txkIfPmw62X5V3Dk39TIe/GvRIs26JA5he5kG2vyg3tve805tKFaSiGOTrhWSHOnZAVPx8tBHen5DQFFzwRzsLGnB7mMxIK4vwZUmi7aA5dR+YYTOkkB9djvhpK6eMBEO6I+WEiraI6nfJZJR3wTNUDE/zdFGPoP50mqFJSs/q6HY/plJWS4VeAXzW1f5cK5ioEksgS+iYgZDMje5pWz5JgO31OnCQsOzsIyzjIO8eyB9ETrTs5h2G7i2jN+j8YDjHYcgqcExSI7SE+kSYt6n5qncc1eaBuc7uDf2RkK5N6cQOM3r891yHRin5f+DsVbtbmKlq2qI+wUA+0JaxT6DW52adZw4sjYMnbuv0gS1Kus= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014)(13003099007)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2025 18:31:32.3313 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9f5934af-697b-4f2a-bd5f-08dd3ca5537e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026369.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6602 From: Ankit Agrawal There is a HW defect on Grace Hopper (GH) to support the Multi-Instance GPU (MIG) feature [1] that necessiated the presence of a 1G region carved out from the device memory and mapped as uncached. The 1G region is shown as a fake BAR (comprising region 2 and 3) to workaround the issue. The Grace Blackwell systems (GB) differ from GH systems in the following aspects: 1. The aforementioned HW defect is fixed on GB systems. 2. There is a usable BAR1 (region 2 and 3) on GB systems for the GPUdirect RDMA feature [2]. This patch accommodate those GB changes by showing the 64b physical device BAR1 (region2 and 3) to the VM instead of the fake one. This takes care of both the differences. Moreover, the entire device memory is exposed on GB as cacheable to the VM as there is no carveout required. Link: https://www.nvidia.com/en-in/technologies/multi-instance-gpu/ [1] Link: https://docs.nvidia.com/cuda/gpudirect-rdma/ [2] Cc: Kevin Tian CC: Jason Gunthorpe Suggested-by: Alex Williamson Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/main.c | 67 +++++++++++++++++++---------- 1 file changed, 45 insertions(+), 22 deletions(-) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index b76368958d1c..778bfd0655de 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -17,9 +17,6 @@ #define RESMEM_REGION_INDEX VFIO_PCI_BAR2_REGION_INDEX #define USEMEM_REGION_INDEX VFIO_PCI_BAR4_REGION_INDEX -/* Memory size expected as non cached and reserved by the VM driver */ -#define RESMEM_SIZE SZ_1G - /* A hardwired and constant ABI value between the GPU FW and VFIO driver. */ #define MEMBLK_SIZE SZ_512M @@ -72,7 +69,7 @@ nvgrace_gpu_memregion(int index, if (index == USEMEM_REGION_INDEX) return &nvdev->usemem; - if (index == RESMEM_REGION_INDEX) + if (nvdev->resmem.memlength && index == RESMEM_REGION_INDEX) return &nvdev->resmem; return NULL; @@ -757,40 +754,67 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev, u64 memphys, u64 memlength) { int ret = 0; + u64 resmem_size = 0; /* - * The VM GPU device driver needs a non-cacheable region to support - * the MIG feature. Since the device memory is mapped as NORMAL cached, - * carve out a region from the end with a different NORMAL_NC - * property (called as reserved memory and represented as resmem). This - * region then is exposed as a 64b BAR (region 2 and 3) to the VM, while - * exposing the rest (termed as usable memory and represented using usemem) - * as cacheable 64b BAR (region 4 and 5). + * On Grace Hopper systems, the VM GPU device driver needs a non-cacheable + * region to support the MIG feature owing to a hardware bug. Since the + * device memory is mapped as NORMAL cached, carve out a region from the end + * with a different NORMAL_NC property (called as reserved memory and + * represented as resmem). This region then is exposed as a 64b BAR + * (region 2 and 3) to the VM, while exposing the rest (termed as usable + * memory and represented using usemem) as cacheable 64b BAR (region 4 and 5). * * devmem (memlength) * |-------------------------------------------------| * | | * usemem.memphys resmem.memphys + * + * This hardware bug is fixed on the Grace Blackwell platforms and the + * presence of the bug can be determined through nvdev->has_mig_hw_bug. + * Thus on systems with the hardware fix, there is no need to partition + * the GPU device memory and the entire memory is usable and mapped as + * NORMAL cached (i.e. resmem size is 0). */ + if (nvdev->has_mig_hw_bug) + resmem_size = SZ_1G; + nvdev->usemem.memphys = memphys; /* * The device memory exposed to the VM is added to the kernel by the - * VM driver module in chunks of memory block size. Only the usable - * memory (usemem) is added to the kernel for usage by the VM - * workloads. Make the usable memory size memblock aligned. + * VM driver module in chunks of memory block size. Note that only the + * usable memory (usemem) is added to the kernel for usage by the VM + * workloads. */ - if (check_sub_overflow(memlength, RESMEM_SIZE, + if (check_sub_overflow(memlength, resmem_size, &nvdev->usemem.memlength)) { ret = -EOVERFLOW; goto done; } /* - * The USEMEM part of the device memory has to be MEMBLK_SIZE - * aligned. This is a hardwired ABI value between the GPU FW and - * VFIO driver. The VM device driver is also aware of it and make - * use of the value for its calculation to determine USEMEM size. + * The usemem region is exposed as a 64B Bar composed of region 4 and 5. + * Calculate and save the BAR size for the region. + */ + nvdev->usemem.bar_size = roundup_pow_of_two(nvdev->usemem.memlength); + + /* + * If the hardware has the fix for MIG, there is no requirement + * for splitting the device memory to create RESMEM. The entire + * device memory is usable and will be USEMEM. Return here for + * such case. + */ + if (!nvdev->has_mig_hw_bug) + goto done; + + /* + * When the device memory is split to workaround the MIG bug on + * Grace Hopper, the USEMEM part of the device memory has to be + * MEMBLK_SIZE aligned. This is a hardwired ABI value between the + * GPU FW and VFIO driver. The VM device driver is also aware of it + * and make use of the value for its calculation to determine USEMEM + * size. Note that the device memory may not be 512M aligned. */ nvdev->usemem.memlength = round_down(nvdev->usemem.memlength, MEMBLK_SIZE); @@ -809,10 +833,9 @@ nvgrace_gpu_init_nvdev_struct(struct pci_dev *pdev, } /* - * The memory regions are exposed as BARs. Calculate and save - * the BAR size for them. + * The resmem region is exposed as a 64b BAR composed of region 2 and 3 + * for Grace Hopper. Calculate and save the BAR size for the region. */ - nvdev->usemem.bar_size = roundup_pow_of_two(nvdev->usemem.memlength); nvdev->resmem.bar_size = roundup_pow_of_two(nvdev->resmem.memlength); done: return ret; From patchwork Fri Jan 24 18:31:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13949824 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2082.outbound.protection.outlook.com [40.107.102.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF78021B90B; Fri, 24 Jan 2025 18:31:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.102.82 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743499; cv=fail; b=Hutqfs9dQHfZqH3TWgw+AWFmx3v+89THx0grYUtTZtHI95s/zyiTZ9GQy0YNP6JndjM0wf5pUE3yTCFl73bqHJ7w0G4xsV9vi1ghzIGexlZd9hV0jiDZsiVC7TjeK0XxyVRw18NH57nkx38wmp2pnHtwhxBtzNwiqvF+0T3/0do= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743499; c=relaxed/simple; bh=+se5AY3kqkj4HP/fRSBua7ksbVwVSZnq8Ak+5V0e/TM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GdPLf+nKFuOoSzH1i2uSepECI4JYV7zu/kvk+PvwCt/Mzjd0HvVA+3nmZTG0nlQqI9tdpBw3kp29G+9o0At5E40hJsmQFTBUis5KCU1Mr5PNdvORwFJDLvMrYnDHfL/Db8Tkp5SqDiOkroI1veen3O5pFcWWkfN+DzS7ACt0DdY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=tt714fY7; arc=fail smtp.client-ip=40.107.102.82 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="tt714fY7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=dNZWWREjY/DQNyMppeQNR0f2webjOOMRNsL3krcGO7W+f9hz1kJVJpqTrXyxSgRpj/TfnBwkTcU7qkaWas1Zuvpoft9iGiXL9fXrFW0Fjztj5hYwp36q9kED0pSpoGnLFf0vTTeqe9mo6/DDE+s62DHNlPYhx0PkncOitwxBXuFLvYNMeOW42Au7Ofm0DZBJr+GjgqYyA9WbLmeFO+FdZWEl+wjW253RulM8qZ36suXKcSLcO+Cy90Y821SdS8zeofKzQC4xdf12w/m4O2Dd2YSlgRoaoCEJ5y3tO4t7O+GnhXbUaHH3axC9gWZVlTwAwzHrg2MlrGA1uXttiMeL0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wtoicHvCWCrwGsqWNInU7GT5fDffHIGfOumosWXmQHM=; b=t1mEQf8X3IuudHjTcLHNWlzEQ1RrvD5s2OjwczVQUEutxlKZMOOb47PDq3e/h4XNvoWfAgGO34Za8Fp9z7yWzIn7WJX/D5v9IgTARFIikPuL832pCDPEHnd+pT6zEKtYSqpndqv+pWxARj94PaSd3eEXmN2AKO3WEmShtLgcODVCeAecjh7bSuhfasSKIaslO28muqSJF1TmK1BsBMj2/CHrKA2oTjGWE0cr3sdM0FLW+vJrB89KQzA7R/I0ZFGRYt/uxiBb9MvrDAFgX93cZu80MPChpyi428bGQOqsGAHQ0BDr6s6r19RiMjghIZKvs8jlXb3yN+PlkpAWf2GpeQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wtoicHvCWCrwGsqWNInU7GT5fDffHIGfOumosWXmQHM=; b=tt714fY7K9zoJtocV99KQ/xp934EU54XnTT0Hp/GU13Uj0AKqvKhNj7eGhgRgMVzmTxIwJ3Xx3aIKSCISXfqHXJDhT9LDtxjuub9WhqrpKjdkiG4o/XWRBMsV8pNhOU9QncccnN3mxqYKjP8yb5sbC2rbhr1dJb41IILaif0Lhc51fX1yazdud/8tEBpyUmzc3z2haQ2J46yzyKyc0NxwgXSJOq7BjjFk5Xch8YXYTz9O+bzJgsg3DpabuOZgQ0+huZMhjoTorNvPq2EvZdCxKEZMtrOYXfAMYKYMrOlCzQ2wY+QxYybOcEKqtj+RadvqJ/AXgzJSwGLQcGAas8FBg== Received: from SA0PR12CA0016.namprd12.prod.outlook.com (2603:10b6:806:6f::21) by PH7PR12MB9224.namprd12.prod.outlook.com (2603:10b6:510:2e7::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.18; Fri, 24 Jan 2025 18:31:33 +0000 Received: from SN1PEPF00026369.namprd02.prod.outlook.com (2603:10b6:806:6f:cafe::4e) by SA0PR12CA0016.outlook.office365.com (2603:10b6:806:6f::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8377.14 via Frontend Transport; Fri, 24 Jan 2025 18:31:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SN1PEPF00026369.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.8 via Frontend Transport; Fri, 24 Jan 2025 18:31:33 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 24 Jan 2025 10:31:16 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Fri, 24 Jan 2025 10:31:16 -0800 Received: from localhost.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14 via Frontend Transport; Fri, 24 Jan 2025 10:31:16 -0800 From: To: , , , , , , CC: , , , , , , , , , , , , , , , , Subject: [PATCH v6 3/4] vfio/nvgrace-gpu: Check the HBM training and C2C link status Date: Fri, 24 Jan 2025 18:31:01 +0000 Message-ID: <20250124183102.3976-4-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250124183102.3976-1-ankita@nvidia.com> References: <20250124183102.3976-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026369:EE_|PH7PR12MB9224:EE_ X-MS-Office365-Filtering-Correlation-Id: 13221090-ac8d-4a90-906b-08dd3ca553f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024|7053199007; X-Microsoft-Antispam-Message-Info: cCQRg94QcmCvowDEc6z5H8zrymUVlBrvrunrF3t4VPtakkKf0H8io+g8GZg9/xnVlqdwETdUixP2X4Yb/VZmjBndoBSd8N5gcnUE4DRC3SnLdFKMTMjtLm579h2sh6jd3UR7g9XfBMuK6WT/O58JvaiJr694thHg19GjDPC0O8KE+1qa+1RClEHdHOZS8jU2TbksNFGZpPw5f3UZXGlMidjFgptvwMxgOTUa/ga7g6zk4N/9KqclYNEZxTU+5Tt+yoMhTgmBGkby21pSfAhfUxIiJZPcd9+HYWgHWp18G7EMWPBjnXgHOllPNNg8Ec+hF25Z5rm9UF5CgyuFw9kOIEyxMkHC5+FA8bM9sGzddm7f48Qa/X/0HFL1Qt8P8CGFG8dnNfPdqNpp8MoPHqRKE3qRKIh6nI5pjXWhgSR6wmyE3E9qGZM4GG32Oh32rKIa8azoBnnuk+qBotLjg95Z4O2E27VmQu5b1xy4FzKKBvB4OyTZzi+XOY8dexndg3nQ79x4yD8Or3M+C+5Kb0Gl+hgAU/zTQL+92WC6LpJvJvk4ibx0+RjA1t4yf229UmOUpVxJt/z29ZjEJ5ErRmW6tBd+DlUMuF2Rs7ceUyje1aK2/DkkkVvEAiSgD/Xi5PaVDWnLwUaMcigUYkdrzUE7TCdagkB+fkJ02IqJ5nEI+JyvltWkhGyF53qu4oSJimSgT+t+wCsq26I3dHsanOSyUaFgBECZ6UGxf3DvIRQTLlwrchrE0v4/BmgpK6TZm0VOT3Z3fnLFYR0763Xt34g9/CWlx2mhPyMP14UDbuUKFwLeD0LPVIMaTS/haYg/ArYX1ke3BivfdF6c6aPBSHFFf8S5mAZt6LrJNdceveBuucxGjEy2P1qzOZRKg1aHGAIGvVN0936B7L1I2owdvIfPhV7cKVoU4CfUFpS2AEFhRABYQzYJuyNOLmKywCNyxw6YTHAvkt9VvUOXkVdhV3g8jRi9iDZvYZ9VIN1N/YyeW6jAkOt/5Kt24rr1//XJiYrmG6itM4fUk51ps7745bet3rgHfOgUjjIWHkA9TkVBOBTgeryTV/mA1qDrYmQPSVnYVk82H0FHHxAIRljK+2Cf8qxEKC2PlLrCCzv+NWIR6hHKnmbxKP4igMt/wfe71nenELVd1jXOUR3OOJGbASLF5NS75pNFxa0oR+10uTC85r+dMZTb4uYwbbxqrxxCK7XYrNNBEkHhH8IUDk+CVG8kKR16C+Fu3IrzwuJs1YvBaiAzGQply0kvXCNTG33dYZFGzOBT8txt7CUPmtejVN5z/ilMlbg0VYNT9TZaBV2JppV5J79/mT6cyK1fgjIyLtBGH1aF0ofiFe8/zylKP8wpkimQkl44bywGEiHcbJSmbtTJ1FEm8CqrvwKZW/kwo6a0ON4WbMMb3HGbzys3YCvC6jxC7xi/ChOdyvcEYQEzlBOufEYadMISDiRxzd5CYauPdmY+cFrxgaZasAfZat+kiyL9WGw8kgNh1fTT0e09Op4= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024)(7053199007);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2025 18:31:33.0813 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 13221090-ac8d-4a90-906b-08dd3ca553f3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026369.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9224 From: Ankit Agrawal In contrast to Grace Hopper systems, the HBM training has been moved out of the UEFI on the Grace Blackwell systems. This reduces the system bootup time significantly. The onus of checking whether the HBM training has completed thus falls on the module. The HBM training status can be determined from a BAR0 register. Similarly, another BAR0 register exposes the status of the CPU-GPU chip-to-chip (C2C) cache coherent interconnect. Based on testing, 30s is determined to be sufficient to ensure initialization completion on all the Grace based systems. Thus poll these register and check for 30s. If the HBM training is not complete or if the C2C link is not ready, fail the probe. While the time is not required on Grace Hopper systems, it is beneficial to make the check to ensure the device is in an expected state. Hence keeping it generalized to both the generations. Ensure that the BAR0 is enabled before accessing the registers. CC: Alex Williamson CC: Kevin Tian CC: Jason Gunthorpe Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/main.c | 72 +++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 778bfd0655de..655a624134cc 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -5,6 +5,8 @@ #include #include +#include +#include /* * The device memory usable to the workloads running in the VM is cached @@ -25,6 +27,13 @@ #define GPU_CAP_DVSEC_REGISTER 3 +#define C2C_LINK_BAR0_OFFSET 0x1498 +#define HBM_TRAINING_BAR0_OFFSET 0x200BC +#define STATUS_READY 0xFF + +#define POLL_QUANTUM_MS 1000 +#define POLL_TIMEOUT_MS (30 * 1000) + /* * The state of the two device memory region - resmem and usemem - is * saved as struct mem_region. @@ -861,6 +870,65 @@ static bool nvgrace_gpu_has_mig_hw_bug(struct pci_dev *pdev) return true; } +/* + * To reduce the system bootup time, the HBM training has + * been moved out of the UEFI on the Grace-Blackwell systems. + * + * The onus of checking whether the HBM training has completed + * thus falls on the module. The HBM training status can be + * determined from a BAR0 register. + * + * Similarly, another BAR0 register exposes the status of the + * CPU-GPU chip-to-chip (C2C) cache coherent interconnect. + * + * Poll these register and check for 30s. If the HBM training is + * not complete or if the C2C link is not ready, fail the probe. + * + * While the wait is not required on Grace Hopper systems, it + * is beneficial to make the check to ensure the device is in an + * expected state. + * + * Ensure that the BAR0 region is enabled before accessing the + * registers. + */ +static int nvgrace_gpu_wait_device_ready(struct pci_dev *pdev) +{ + unsigned long timeout = jiffies + msecs_to_jiffies(POLL_TIMEOUT_MS); + void __iomem *io; + int ret = -ETIME; + + ret = pci_enable_device(pdev); + if (ret) + return ret; + + ret = pci_request_selected_regions(pdev, 1 << 0, KBUILD_MODNAME); + if (ret) + goto request_region_exit; + + io = pci_iomap(pdev, 0, 0); + if (!io) { + ret = -ENOMEM; + goto iomap_exit; + } + + do { + if ((ioread32(io + C2C_LINK_BAR0_OFFSET) == STATUS_READY) && + (ioread32(io + HBM_TRAINING_BAR0_OFFSET) == STATUS_READY)) { + ret = 0; + goto reg_check_exit; + } + msleep(POLL_QUANTUM_MS); + } while (!time_after(jiffies, timeout)); + +reg_check_exit: + pci_iounmap(pdev, io); +iomap_exit: + pci_release_selected_regions(pdev, 1 << 0); +request_region_exit: + pci_disable_device(pdev); + return ret; +} + static int nvgrace_gpu_probe(struct pci_dev *pdev, const struct pci_device_id *id) { @@ -869,6 +937,10 @@ static int nvgrace_gpu_probe(struct pci_dev *pdev, u64 memphys, memlength; int ret; + ret = nvgrace_gpu_wait_device_ready(pdev); + if (ret) + return ret; + ret = nvgrace_gpu_fetch_memory_property(pdev, &memphys, &memlength); if (!ret) ops = &nvgrace_gpu_pci_ops; From patchwork Fri Jan 24 18:31:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankit Agrawal X-Patchwork-Id: 13949827 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2086.outbound.protection.outlook.com [40.107.236.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9874421C178; Fri, 24 Jan 2025 18:31:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.86 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743501; cv=fail; b=lgE/kgMQnkTmorL3YrwFJmAD0e6+XBqSS24EOlkg6KZk/jg8PPvqDAnC77rklI/Q/MmMGzMMGVVMHr0Tc5U7mupLbPSc5NK7m4YZmwL2XGDCKQZMzluGNU0jQdxJB/yuSOa1Fy7bYyJ34e9yHE7euJ/sZmO8vKwxm1Y6keDC7N4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737743501; c=relaxed/simple; bh=YDcvHxq2botZi81D6AEE9wbFhLPHVtZeJaVsIZtR6o4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tWK03GQt5Q0jziK+k/D1l7vsJlqAUtC999H24nz30CQVz0zm52tLuu5W108Pq4Hu752FaXvlSBveKBpS6/eUDq7OKUSXPNV9cmhFKXmovYGN2O+RkE7ToY00OJZb71vypDVJ9Jt2wlDgCEcLW+yteKcfQy838EN6pp0IC2boZO8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ewsMC/du; arc=fail smtp.client-ip=40.107.236.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ewsMC/du" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vUMcs2utO4c/uCXSqgNWfJR5Tx2G/7IXLm2yR+C7GfGJSSy7VKgkmfmZGFXS1RWlpM3iOwlvoU5/YUmyU6kawnAsgunCDSQCkSlfKiQhQNOP8G9+6dtJnwAIj19ZI3wpc5p/Kax3MxcXMOwFDak2Rc3jKisn9NquNbxKonObGTGpUqsqsygMgUfNgHIGc37ko51TP3RicqS+Z+RYd5NApFckLbgRVB+O+l4WKUtc1B9tlWDilIlfutsSaa8VmR8zNNkN/IoW+Cw0icmE1XZ7Y85PdjkvbaZajRdW5ZZr4AvdOlscZsBMGuSe8G0AfWzNoDPARL8CmLNvyEle3AE7ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hDUePpC+uBA5jmjoJiISfzCOb5fJDGl6q2+V7qsG1QU=; b=JJA4rIRqZaWhtkVsQxKA9Tq+gGxUpOOlcayA9wMx6NB4FMXSLUtnhd2dWfgy8UMZWIapnkZViAuq61Hr0YaR33aYLCygyRqfpIM+vyWZoy5eJ+Ca2UokAu/zPfBLuy0pAMYleVK0TVl6QcSXMuiIfh1Mt+uaJR/46Dv4DJ6LUFBkKftIqY+oTLZLnSFpU/H7iq/Q2EdzOGfmSY0uCFsZSM2rwNQ85BnUQm1Fxj7/FOOPpZDwXIZAwsZeQL7ukKYT/eD6+jnpLqEE1zxD4LqJVsDVXdULVgpexidOrs80IA7OSPEjZvJz2bQephAlf6IwjkWGnNlJlTAyhrayqiTnRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hDUePpC+uBA5jmjoJiISfzCOb5fJDGl6q2+V7qsG1QU=; b=ewsMC/duVr1IXfk3JV20MvT3H1x1ZFo43J6m/rrjLNuB2YqHunfkMjZ4Tue1YUt1QzrKVRCAL4j3ei+vThB0ocO4bNES8C73s3E+BcA15e/gK6bGrHAOQJzHqpzlW3O/tCRaoGy8G3+cmFcMUrHPFcw9/tGVYqGF9S41XpSjDSFqYn/xuOELQTOgwWm55BUvi4Os93csY7jnmJAHSmVNKGrivDufYeZWaPEQZyQXM9qckkoMEFR4/AVVjPaf2u/TZ8BzsNYKmRUJlWi0DhjLjfG+WF1waySKRWqaozoCWu+N1+hfHE2FhB2lJnyHS0+ZGdQrIFlj7Z+dMWtXppHQMw== Received: from MN2PR05CA0050.namprd05.prod.outlook.com (2603:10b6:208:236::19) by DS0PR12MB8413.namprd12.prod.outlook.com (2603:10b6:8:f9::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.19; Fri, 24 Jan 2025 18:31:33 +0000 Received: from BL6PEPF0001AB74.namprd02.prod.outlook.com (2603:10b6:208:236:cafe::f0) by MN2PR05CA0050.outlook.office365.com (2603:10b6:208:236::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8377.13 via Frontend Transport; Fri, 24 Jan 2025 18:31:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF0001AB74.mail.protection.outlook.com (10.167.242.167) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8377.8 via Frontend Transport; Fri, 24 Jan 2025 18:31:33 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 24 Jan 2025 10:31:17 -0800 Received: from drhqmail201.nvidia.com (10.126.190.180) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14; Fri, 24 Jan 2025 10:31:17 -0800 Received: from localhost.nvidia.com (10.127.8.14) by mail.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.14 via Frontend Transport; Fri, 24 Jan 2025 10:31:17 -0800 From: To: , , , , , , CC: , , , , , , , , , , , , , , , , Subject: [PATCH v6 4/4] vfio/nvgrace-gpu: Add GB200 SKU to the devid table Date: Fri, 24 Jan 2025 18:31:02 +0000 Message-ID: <20250124183102.3976-5-ankita@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250124183102.3976-1-ankita@nvidia.com> References: <20250124183102.3976-1-ankita@nvidia.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: AnonymousSubmission X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB74:EE_|DS0PR12MB8413:EE_ X-MS-Office365-Filtering-Correlation-Id: 5c3d0532-6828-4474-2dbc-08dd3ca553fa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|376014|36860700013; X-Microsoft-Antispam-Message-Info: h93Ou6q2TDl70kH4HqiVP3DFm37tzj0vcQLSYWotf5x4H91/bdITOC5h6bURcpf3g/M1QJbi/XrlEUodvVsJD3EajVWDRJWt0dsNFWK+p1xyXWjQtELK9V9AhnOHag7XgFg6auN+JC3/hd6YWAw10xG1OiZut2DzmpGKxZhj7YiMOlzpvtbR76cQgkD69v3Va4cwKt2TZ0yJ/3LiKN97konyoCdFXtIuZMi4KFhnnebnOD+oNHRQtq2FjO19cuKk/CO+ajas+CyWsDuj4txyZEq1HxCeapfe50VDZEOA1N0OgkwacSjjctJB/8Pwy8r4A0mdF5uq+eSYCb1n8ETEoBw2FirkBLlaf295dcMtPggRbMBS2aln4An4adUCxpSgUZGbdp2ArS2awYRxU9PWbUwCyPP/hNIEMrYrSk+vXgNpa1kqm5iba9WEMJN0JyuwCS9pE7NqKsq6WQsoEI1MsC7emfT0+dTNbSNHZXqwLZhLD0GS0OWLtTlIzLjf+Arb0M7dUfgb5h8JOharfdhVC0FJUxMKiAeOOvtrsVblMIx7pWihChEp1RZGSKxzmAmD/MrBaX1m7odQieI0ORKeLYok8AVqi+2p/ESvlFxnTsiL/sa3oFOA9gd2rPqsirZE+6CszVsPFxAYz4aBIIh2m6XkkHR3oJESbUGBA0wgphcuSwxVuI5cUC9R+r+FSR52edeMoXahMIECybV+4YrKqzB4rsLrk+oiG+WsjlWGeBpS3r+zZcPWByqWaR0NySpAS4uQzpZYpIw7VAlTjZLPX3WkxxMW1H5+N2QC7uvoPjrAo1MO1mZCsijVB1unJsoJtFAsq7ECtiExeFXbQxeeJb05e4Ks/mLm1iH1YMqri0i9PQ76aioArfZ392LhYiwm+50sl/C8wsZiqznXYxtghNXP3ZYQc68mPgaHBy9oGGaim5maHCb8mKv8WFBxzkNtzkclxpMekEUQZh7ObhFB+VM+SINWZ5Sj5g08Wf+qnr2CxwjEtyKtAZjPgk8wGD04KfLQF/++juIJbeXlBW8AqIWCY8IIV6QH7QhE6BoZ1mQBIJlBJzCxrj2bMY428wIHvQpNvvopO50ewaNPCYzpqd/uDjwAfcdLFZGB49TDUzH2S+j1is5EghmWRK9DXvYYF/OWaCTlw+9gYVRtG3H6bmJlXwTkTW8Ual3Fj1UZAB/q6Gr07rsEQn3PKHqmaM1m+TuSPHquBzS0QEXtd6DJ/KPp9z6KR73S5OHV5huWfh/qWOZqh5r9b6ZJTDNt/BcCcoTJ/j2AJlrFryFPKKwm6Ss+csavxpaoSV8Y8KP6mgXF3ZocRfXTrjBesjkFL8YSUPeRMVOFQwCVhXeW6wBkAB1t/t2lzP/s5CqsEJmYS1vmoPaHDk7Iu8c7+RuDzki+R/s4ZQzuCQik/3UeW73ydqC1RRF/x6OyYJp5N2L62P+tkgwx8ri4wol+CicuEOE9FECWBc/JGjDMO9bRT4jKgpTiToM0qof3ItmamQHARyY= X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2025 18:31:33.1247 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5c3d0532-6828-4474-2dbc-08dd3ca553fa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB74.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8413 From: Ankit Agrawal NVIDIA is productizing the new Grace Blackwell superchip SKU bearing device ID 0x2941. Add the SKU devid to nvgrace_gpu_vfio_pci_table. CC: Alex Williamson Signed-off-by: Ankit Agrawal --- drivers/vfio/pci/nvgrace-gpu/main.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 655a624134cc..e5ac39c4cc6b 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -991,6 +991,8 @@ static const struct pci_device_id nvgrace_gpu_vfio_pci_table[] = { { PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2345) }, /* GH200 SKU */ { PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2348) }, + /* GB200 SKU */ + { PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_NVIDIA, 0x2941) }, {} };