From patchwork Wed Jul 21 16:16:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 12391545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_NONE, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A683C636CA for ; Wed, 21 Jul 2021 16:17:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E43196120E for ; Wed, 21 Jul 2021 16:17:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233643AbhGUPhT (ORCPT ); Wed, 21 Jul 2021 11:37:19 -0400 Received: from mail-mw2nam12on2060.outbound.protection.outlook.com ([40.107.244.60]:16512 "EHLO NAM12-MW2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S233326AbhGUPhR (ORCPT ); Wed, 21 Jul 2021 11:37:17 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VhtThqzZnDXGUuIT+tFhcV75gnjuWJL+wgRHTYMGmVInVxT/aUDXPGk+ZVrOH+pmvV2Va6rlJ04XnohZ+W42u3UVYfRwVYxEkrUMfdfzbX+KsSqpBFoawKsEqGjz9+Lj4s4SoY6+WLut1JBB7oalQ1OpLr5GCGmsETOOp8z5IDVswnI4nRgUSnhrylLzO1UN6jTmle92THOs5g6l35Vz3l2fc2scAd2354ryYvaDNieskesslMv8OJD/8sGs2zp7bJ6Z//zNGIBomVyt7mY2Q6BkpbXNJivE2RXMEkKZCrgJL7JbH+nEwF1aq1sxdhOK4ul/u6qfQJL2HH3R2++7BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aIagYfY6GgpW1XK8x3x1GRI44tGKgkel0ogECKQjq7k=; b=X+GRhUnhsp+5QzOSHW0HAs5zoiUO2Ym1/qbSCdR3KpFGl7Nx7GUMuEisRvDgjQePTFhhCHRogZvKu8zV42ay4nGNjXsljTDDR7wP8EUGqK1ycudVnNMvAC66t0vltNY0tYsXcrLtbGdxl+yUeA30Np4tQ46sjnhK/nnGhlGl34AoFpJIw9tBaVdIMXSpofOP8frJqkPdAjewyf23AS4HHQsQLrDh51LfruAN2zkOdUEmWE1n+HLZWospS8qt5vXh8or0J8diiCEUklCfLWYVaCjS2EhodHBkU9piPaCg2eXrMfxsrv/XsvOdOtrdY5Jz7MhRDoh3/LGMbSmeFzbecQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aIagYfY6GgpW1XK8x3x1GRI44tGKgkel0ogECKQjq7k=; b=C4SG47Yq9pQpl6UdwXE9iZqA45zPvmnmHJcnyL6itpS2M0VpE4dN21+tjWBR5Bs8EZQxTebuhIq8xPziGyC1g05vnZEqu+J72MPRda7XKzIcw1WCRlF+X5dMAu0T15ITV5T62sWbqRcGLDRlvwpq81f7WK2TBEuHqNVRX5ybeUosaYnc0jnjm657b2WXYjHVoOgyv8BS3vdVcKRJKbci1DaueBz/jBuF1/sLyvnseY+4b0SI+p9jYHXuP1KHFULXSjq8a5s/jQIZLDC0XfvLXh4Hr/hC+9x21GEe9HXR0axlfmPBKZJHIRepG+/tbfPB7T4GTD9k+Je8QppC9E3hyg== Received: from DM6PR01CA0015.prod.exchangelabs.com (2603:10b6:5:296::20) by BN6PR12MB1122.namprd12.prod.outlook.com (2603:10b6:404:20::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.29; Wed, 21 Jul 2021 16:17:48 +0000 Received: from DM6NAM11FT067.eop-nam11.prod.protection.outlook.com (2603:10b6:5:296:cafe::d7) by DM6PR01CA0015.outlook.office365.com (2603:10b6:5:296::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Wed, 21 Jul 2021 16:17:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT067.mail.protection.outlook.com (10.13.172.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4352.24 via Frontend Transport; Wed, 21 Jul 2021 16:17:48 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 21 Jul 2021 09:17:47 -0700 Received: from vdi.nvidia.com (172.20.187.6) by mail.nvidia.com (172.20.187.12) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 21 Jul 2021 16:17:43 +0000 From: Yishai Hadas To: , , , , , , , CC: , , , , , , , , , Subject: [PATCH 06/12] vfio/pci: Split the pci_driver code out of vfio_pci_core.c Date: Wed, 21 Jul 2021 19:16:03 +0300 Message-ID: <20210721161609.68223-7-yishaih@nvidia.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20210721161609.68223-1-yishaih@nvidia.com> References: <20210721161609.68223-1-yishaih@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aa1c5093-27e3-449c-6b40-08d94c6314d5 X-MS-TrafficTypeDiagnostic: BN6PR12MB1122: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3383; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LBHen/xoRbJvXaAzkRd0WGxarWI/I76Pa/2ap/0YacMWAFHRE84ULPfO6A0XpkZUrP3Ir+oRC24qbrbAZ2zkIixQ/acndn1Bcx0amxulxAC3FJ14xCrqQrW4bSVxoj66xfAngKaoJ7u1YeH2VdbLYLuaDLG3ukpg6W2LN5/j8YPLR5NAtuTtrAkFJhZRZLW3bn7cL67PJL8iWsAygzpCL4pWBB3o1Hf2y7jk4IJZbYrP7AI5sFwHXLKA7GFLSJzhFyy5P85naT+H4kU8JkNSboo/UREf+GsH/EKTrWNJez7O8za2MwfUbjg0fTMnoYk+TV/9kmXlm9vLQp/jwQ6/N2MZfqVzGaHmei5NzSjz/I+3mo7VhzJaQ+zRgNZpA6rL0mSPfsnXB7bIZyjfmYz1i/qe/vHOgBVAbh8v47JUOl6dOG7yOLkSLE33WvuUyhHF0r5fcmFKqHu6wJQTx0omCyrnQ/varBy4U+bnTt6y249ezoeNdfi/ZFT2NNIF278w2upWyUoXZ3ZzFU9AfcGBuvi3Ko62jWUk4zWgvS4dBLTd8NF7CYBZL8H+5kB4y1puW8lJ0gBO/JH6dTE7t9GuFlIViCrc95i2cAs032wTWHfc+52E+8EPb1Ejup+niDl0aWzsl5d34xYO+etTBrS6iIr0KErLtB99Rso2zJR8K8Jc0yWGxZlgrmpA7nzwdSfyj0ZyWVTHg2g6FmNrAtgAiYbCmdD1Y/UKZ0F/vXyYbkg= X-Forefront-Antispam-Report: CIP:216.228.112.32;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:schybrid01.nvidia.com;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(498600001)(83380400001)(82310400003)(30864003)(8676002)(7416002)(8936002)(4326008)(36756003)(86362001)(7696005)(70586007)(110136005)(70206006)(336012)(2616005)(54906003)(2906002)(47076005)(5660300002)(26005)(426003)(6666004)(107886003)(1076003)(7636003)(356005)(186003)(36860700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 16:17:48.3060 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: aa1c5093-27e3-449c-6b40-08d94c6314d5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.32];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT067.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1122 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Max Gurtovoy Split the vfio_pci driver into two logical parts, the 'struct pci_driver' (vfio_pci.c) which implements "Generic VFIO support for any PCI device" and a library of code (vfio_pci_core.c) that helps implementing a struct vfio_device on top of a PCI device. vfio_pci.ko continues to present the same interface under sysfs and this change should have no functional impact. Following patches will turn vfio_pci and vfio_pci_core into a separate module. This is a preparation for allowing another module to provide the pci_driver and allow that module to customize how VFIO is setup, inject its own operations, and easily extend vendor specific functionality. At this point the vfio_pci_core still contains a lot of vfio_pci functionality mixed into it. Following patches will move more of the large scale items out, but another cleanup series will be needed to get everything. Signed-off-by: Max Gurtovoy Signed-off-by: Jason Gunthorpe Signed-off-by: Yishai Hadas --- drivers/vfio/pci/Makefile | 2 +- drivers/vfio/pci/vfio_pci.c | 227 ++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 264 +++++++------------------------ drivers/vfio/pci/vfio_pci_core.h | 23 +++ 4 files changed, 305 insertions(+), 211 deletions(-) create mode 100644 drivers/vfio/pci/vfio_pci.c diff --git a/drivers/vfio/pci/Makefile b/drivers/vfio/pci/Makefile index 66a40488e967..8aa517b4b671 100644 --- a/drivers/vfio/pci/Makefile +++ b/drivers/vfio/pci/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -vfio-pci-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o +vfio-pci-y := vfio_pci.o vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o vfio-pci-$(CONFIG_S390) += vfio_pci_zdev.o diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c new file mode 100644 index 000000000000..4ccfbac0797a --- /dev/null +++ b/drivers/vfio/pci/vfio_pci.c @@ -0,0 +1,227 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved + * + * Copyright (C) 2012 Red Hat, Inc. All rights reserved. + * Author: Alex Williamson + * + * Derived from original vfio: + * Copyright 2010 Cisco Systems, Inc. All rights reserved. + * Author: Tom Lyon, pugs@cisco.com + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vfio_pci_core.h" + +#define DRIVER_VERSION "0.2" +#define DRIVER_AUTHOR "Alex Williamson " +#define DRIVER_DESC "VFIO PCI - User Level meta-driver" + +static char ids[1024] __initdata; +module_param_string(ids, ids, sizeof(ids), 0); +MODULE_PARM_DESC(ids, "Initial PCI IDs to add to the vfio driver, format is \"vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]\" and multiple comma separated entries can be specified"); + +static bool enable_sriov; +#ifdef CONFIG_PCI_IOV +module_param(enable_sriov, bool, 0644); +MODULE_PARM_DESC(enable_sriov, "Enable support for SR-IOV configuration. Enabling SR-IOV on a PF typically requires support of the userspace PF driver, enabling VFs without such support may result in non-functional VFs or PF."); +#endif + +static bool disable_denylist; +module_param(disable_denylist, bool, 0444); +MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users."); + +static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev) +{ + switch (pdev->vendor) { + case PCI_VENDOR_ID_INTEL: + switch (pdev->device) { + case PCI_DEVICE_ID_INTEL_QAT_C3XXX: + case PCI_DEVICE_ID_INTEL_QAT_C3XXX_VF: + case PCI_DEVICE_ID_INTEL_QAT_C62X: + case PCI_DEVICE_ID_INTEL_QAT_C62X_VF: + case PCI_DEVICE_ID_INTEL_QAT_DH895XCC: + case PCI_DEVICE_ID_INTEL_QAT_DH895XCC_VF: + return true; + default: + return false; + } + } + + return false; +} + +static bool vfio_pci_is_denylisted(struct pci_dev *pdev) +{ + if (!vfio_pci_dev_in_denylist(pdev)) + return false; + + if (disable_denylist) { + pci_warn(pdev, + "device denylist disabled - allowing device %04x:%04x.\n", + pdev->vendor, pdev->device); + return false; + } + + pci_warn(pdev, "%04x:%04x exists in vfio-pci device denylist, driver probing disallowed.\n", + pdev->vendor, pdev->device); + + return true; +} + +static const struct vfio_device_ops vfio_pci_ops = { + .name = "vfio-pci", + .open_device = vfio_pci_core_open_device, + .close_device = vfio_pci_core_close_device, + .ioctl = vfio_pci_core_ioctl, + .read = vfio_pci_core_read, + .write = vfio_pci_core_write, + .mmap = vfio_pci_core_mmap, + .request = vfio_pci_core_request, + .match = vfio_pci_core_match, +}; + +static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct vfio_pci_core_device *vdev; + int ret; + + if (vfio_pci_is_denylisted(pdev)) + return -EINVAL; + + vdev = kzalloc(sizeof(*vdev), GFP_KERNEL); + if (!vdev) + return -ENOMEM; + vfio_pci_core_init_device(vdev, pdev, &vfio_pci_ops); + + ret = vfio_pci_core_register_device(vdev); + if (ret) + goto out_free; + return 0; + +out_free: + vfio_pci_core_uninit_device(vdev); + kfree(vdev); + return ret; +} + +static void vfio_pci_remove(struct pci_dev *pdev) +{ + struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); + + vfio_pci_core_unregister_device(vdev); + vfio_pci_core_uninit_device(vdev); + kfree(vdev); +} + +static int vfio_pci_sriov_configure(struct pci_dev *pdev, int nr_virtfn) +{ + might_sleep(); + + if (!enable_sriov) + return -ENOENT; + + return vfio_pci_core_sriov_configure(pdev, nr_virtfn); +} + +static struct pci_driver vfio_pci_driver = { + .name = "vfio-pci", + .id_table = NULL, /* only dynamic ids */ + .probe = vfio_pci_probe, + .remove = vfio_pci_remove, + .sriov_configure = vfio_pci_sriov_configure, + .err_handler = &vfio_pci_core_err_handlers, +}; + +static void __init vfio_pci_fill_ids(void) +{ + char *p, *id; + int rc; + + /* no ids passed actually */ + if (ids[0] == '\0') + return; + + /* add ids specified in the module parameter */ + p = ids; + while ((id = strsep(&p, ","))) { + unsigned int vendor, device, subvendor = PCI_ANY_ID, + subdevice = PCI_ANY_ID, class = 0, class_mask = 0; + int fields; + + if (!strlen(id)) + continue; + + fields = sscanf(id, "%x:%x:%x:%x:%x:%x", + &vendor, &device, &subvendor, &subdevice, + &class, &class_mask); + + if (fields < 2) { + pr_warn("invalid id string \"%s\"\n", id); + continue; + } + + rc = pci_add_dynid(&vfio_pci_driver, vendor, device, + subvendor, subdevice, class, class_mask, 0); + if (rc) + pr_warn("failed to add dynamic id [%04x:%04x[%04x:%04x]] class %#08x/%08x (%d)\n", + vendor, device, subvendor, subdevice, + class, class_mask, rc); + else + pr_info("add [%04x:%04x[%04x:%04x]] class %#08x/%08x\n", + vendor, device, subvendor, subdevice, + class, class_mask); + } +} + +static int __init vfio_pci_init(void) +{ + int ret; + + ret = vfio_pci_core_init(); + if (ret) + return ret; + + /* Register and scan for devices */ + ret = pci_register_driver(&vfio_pci_driver); + if (ret) + goto out; + + vfio_pci_fill_ids(); + + if (disable_denylist) + pr_warn("device denylist disabled.\n"); + + return 0; + +out: + vfio_pci_core_cleanup(); + return ret; +} +module_init(vfio_pci_init); + +static void __exit vfio_pci_cleanup(void) +{ + pci_unregister_driver(&vfio_pci_driver); + vfio_pci_core_cleanup(); +} +module_exit(vfio_pci_cleanup); + +MODULE_VERSION(DRIVER_VERSION); +MODULE_LICENSE("GPL v2"); +MODULE_AUTHOR(DRIVER_AUTHOR); +MODULE_DESCRIPTION(DRIVER_DESC); diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 99f579c23ddd..8323acc5d3b7 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -8,8 +8,6 @@ * Author: Tom Lyon, pugs@cisco.com */ -#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt - #include #include #include @@ -29,14 +27,6 @@ #include "vfio_pci_core.h" -#define DRIVER_VERSION "0.2" -#define DRIVER_AUTHOR "Alex Williamson " -#define DRIVER_DESC "VFIO PCI - User Level meta-driver" - -static char ids[1024] __initdata; -module_param_string(ids, ids, sizeof(ids), 0); -MODULE_PARM_DESC(ids, "Initial PCI IDs to add to the vfio driver, format is \"vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]\" and multiple comma separated entries can be specified"); - static bool nointxmask; module_param_named(nointxmask, nointxmask, bool, S_IRUGO | S_IWUSR); MODULE_PARM_DESC(nointxmask, @@ -53,16 +43,6 @@ module_param(disable_idle_d3, bool, S_IRUGO | S_IWUSR); MODULE_PARM_DESC(disable_idle_d3, "Disable using the PCI D3 low power state for idle, unused devices"); -static bool enable_sriov; -#ifdef CONFIG_PCI_IOV -module_param(enable_sriov, bool, 0644); -MODULE_PARM_DESC(enable_sriov, "Enable support for SR-IOV configuration. Enabling SR-IOV on a PF typically requires support of the userspace PF driver, enabling VFs without such support may result in non-functional VFs or PF."); -#endif - -static bool disable_denylist; -module_param(disable_denylist, bool, 0444); -MODULE_PARM_DESC(disable_denylist, "Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users."); - static inline bool vfio_vga_disabled(void) { #ifdef CONFIG_VFIO_PCI_VGA @@ -72,44 +52,6 @@ static inline bool vfio_vga_disabled(void) #endif } -static bool vfio_pci_dev_in_denylist(struct pci_dev *pdev) -{ - switch (pdev->vendor) { - case PCI_VENDOR_ID_INTEL: - switch (pdev->device) { - case PCI_DEVICE_ID_INTEL_QAT_C3XXX: - case PCI_DEVICE_ID_INTEL_QAT_C3XXX_VF: - case PCI_DEVICE_ID_INTEL_QAT_C62X: - case PCI_DEVICE_ID_INTEL_QAT_C62X_VF: - case PCI_DEVICE_ID_INTEL_QAT_DH895XCC: - case PCI_DEVICE_ID_INTEL_QAT_DH895XCC_VF: - return true; - default: - return false; - } - } - - return false; -} - -static bool vfio_pci_is_denylisted(struct pci_dev *pdev) -{ - if (!vfio_pci_dev_in_denylist(pdev)) - return false; - - if (disable_denylist) { - pci_warn(pdev, - "device denylist disabled - allowing device %04x:%04x.\n", - pdev->vendor, pdev->device); - return false; - } - - pci_warn(pdev, "%04x:%04x exists in vfio-pci device denylist, driver probing disallowed.\n", - pdev->vendor, pdev->device); - - return true; -} - /* * Our VGA arbiter participation is limited since we don't know anything * about the device itself. However, if the device is the only VGA device @@ -497,8 +439,6 @@ static void vfio_pci_disable(struct vfio_pci_core_device *vdev) vfio_pci_set_power_state(vdev, PCI_D3hot); } -static struct pci_driver vfio_pci_driver; - static struct vfio_pci_core_device *get_pf_vdev(struct vfio_pci_core_device *vdev) { struct pci_dev *physfn = pci_physfn(vdev->pdev); @@ -511,7 +451,7 @@ static struct vfio_pci_core_device *get_pf_vdev(struct vfio_pci_core_device *vde if (!pf_dev) return NULL; - if (pci_dev_driver(physfn) != &vfio_pci_driver) { + if (pci_dev_driver(physfn) != pci_dev_driver(vdev->pdev)) { vfio_device_put(pf_dev); return NULL; } @@ -534,7 +474,7 @@ static void vfio_pci_vf_token_user_add(struct vfio_pci_core_device *vdev, int va vfio_device_put(&pf_vdev->vdev); } -static void vfio_pci_core_close_device(struct vfio_device *core_vdev) +void vfio_pci_core_close_device(struct vfio_device *core_vdev) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -555,7 +495,7 @@ static void vfio_pci_core_close_device(struct vfio_device *core_vdev) mutex_unlock(&vdev->igate); } -static int vfio_pci_core_open_device(struct vfio_device *core_vdev) +int vfio_pci_core_open_device(struct vfio_device *core_vdev) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -728,8 +668,8 @@ int vfio_pci_register_dev_region(struct vfio_pci_core_device *vdev, return 0; } -static long vfio_pci_core_ioctl(struct vfio_device *core_vdev, - unsigned int cmd, unsigned long arg) +long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, + unsigned long arg) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1303,8 +1243,8 @@ static ssize_t vfio_pci_rw(struct vfio_pci_core_device *vdev, char __user *buf, return -EINVAL; } -static ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, - size_t count, loff_t *ppos) +ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, + size_t count, loff_t *ppos) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1315,8 +1255,8 @@ static ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *bu return vfio_pci_rw(vdev, buf, count, ppos, false); } -static ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf, - size_t count, loff_t *ppos) +ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf, + size_t count, loff_t *ppos) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1534,7 +1474,7 @@ static const struct vm_operations_struct vfio_pci_mmap_ops = { .fault = vfio_pci_mmap_fault, }; -static int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma) +int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1605,7 +1545,7 @@ static int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_stru return 0; } -static void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count) +void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1721,7 +1661,7 @@ static int vfio_pci_validate_vf_token(struct vfio_pci_core_device *vdev, #define VF_TOKEN_ARG "vf_token=" -static int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf) +int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf) { struct vfio_pci_core_device *vdev = container_of(core_vdev, struct vfio_pci_core_device, vdev); @@ -1771,18 +1711,6 @@ static int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf) return 1; /* Match */ } -static const struct vfio_device_ops vfio_pci_ops = { - .name = "vfio-pci", - .open_device = vfio_pci_core_open_device, - .close_device = vfio_pci_core_close_device, - .ioctl = vfio_pci_core_ioctl, - .read = vfio_pci_core_read, - .write = vfio_pci_core_write, - .mmap = vfio_pci_core_mmap, - .request = vfio_pci_core_request, - .match = vfio_pci_core_match, -}; - static int vfio_pci_bus_notifier(struct notifier_block *nb, unsigned long action, void *data) { @@ -1797,12 +1725,12 @@ static int vfio_pci_bus_notifier(struct notifier_block *nb, pci_info(vdev->pdev, "Captured SR-IOV VF %s driver_override\n", pci_name(pdev)); pdev->driver_override = kasprintf(GFP_KERNEL, "%s", - vfio_pci_ops.name); + vdev->vdev.ops->name); } else if (action == BUS_NOTIFY_BOUND_DRIVER && pdev->is_virtfn && physfn == vdev->pdev) { struct pci_driver *drv = pci_dev_driver(pdev); - if (drv && drv != &vfio_pci_driver) + if (drv && drv != pci_dev_driver(vdev->pdev)) pci_warn(vdev->pdev, "VF %s bound to driver %s while PF bound to vfio-pci\n", pci_name(pdev), drv->name); @@ -1873,15 +1801,39 @@ static void vfio_pci_vga_uninit(struct vfio_pci_core_device *vdev) VGA_RSRC_LEGACY_MEM); } -static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) +void vfio_pci_core_init_device(struct vfio_pci_core_device *vdev, + struct pci_dev *pdev, + const struct vfio_device_ops *vfio_pci_ops) { - struct vfio_pci_core_device *vdev; + vfio_init_group_dev(&vdev->vdev, &pdev->dev, vfio_pci_ops); + vdev->pdev = pdev; + vdev->irq_type = VFIO_PCI_NUM_IRQS; + mutex_init(&vdev->igate); + spin_lock_init(&vdev->irqlock); + mutex_init(&vdev->ioeventfds_lock); + INIT_LIST_HEAD(&vdev->dummy_resources_list); + INIT_LIST_HEAD(&vdev->ioeventfds_list); + mutex_init(&vdev->vma_lock); + INIT_LIST_HEAD(&vdev->vma_list); + init_rwsem(&vdev->memory_lock); +} + +void vfio_pci_core_uninit_device(struct vfio_pci_core_device *vdev) +{ + mutex_destroy(&vdev->igate); + mutex_destroy(&vdev->ioeventfds_lock); + mutex_destroy(&vdev->vma_lock); + vfio_uninit_group_dev(&vdev->vdev); + kfree(vdev->region); + kfree(vdev->pm_save); +} + +int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) +{ + struct pci_dev *pdev = vdev->pdev; struct iommu_group *group; int ret; - if (vfio_pci_is_denylisted(pdev)) - return -EINVAL; - if (pdev->hdr_type != PCI_HEADER_TYPE_NORMAL) return -EINVAL; @@ -1902,24 +1854,6 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) if (!group) return -EINVAL; - vdev = kzalloc(sizeof(*vdev), GFP_KERNEL); - if (!vdev) { - ret = -ENOMEM; - goto out_group_put; - } - - vfio_init_group_dev(&vdev->vdev, &pdev->dev, &vfio_pci_ops); - vdev->pdev = pdev; - vdev->irq_type = VFIO_PCI_NUM_IRQS; - mutex_init(&vdev->igate); - spin_lock_init(&vdev->irqlock); - mutex_init(&vdev->ioeventfds_lock); - INIT_LIST_HEAD(&vdev->dummy_resources_list); - INIT_LIST_HEAD(&vdev->ioeventfds_list); - mutex_init(&vdev->vma_lock); - INIT_LIST_HEAD(&vdev->vma_list); - init_rwsem(&vdev->memory_lock); - if (pci_is_root_bus(pdev->bus)) ret = vfio_assign_device_set(&vdev->vdev, vdev); else if (!pci_probe_reset_slot(pdev->slot)) @@ -1927,10 +1861,10 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) else ret = vfio_assign_device_set(&vdev->vdev, pdev->bus); if (ret) - goto out_uninit; + goto out_group_put; ret = vfio_pci_vf_init(vdev); if (ret) - goto out_uninit; + goto out_group_put; ret = vfio_pci_vga_init(vdev); if (ret) goto out_vf; @@ -1962,36 +1896,26 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) vfio_pci_set_power_state(vdev, PCI_D0); out_vf: vfio_pci_vf_uninit(vdev); -out_uninit: - vfio_uninit_group_dev(&vdev->vdev); - kfree(vdev->pm_save); - kfree(vdev); out_group_put: vfio_iommu_group_put(group, &pdev->dev); return ret; } -static void vfio_pci_remove(struct pci_dev *pdev) +void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev) { - struct vfio_pci_core_device *vdev = dev_get_drvdata(&pdev->dev); + struct pci_dev *pdev = vdev->pdev; pci_disable_sriov(pdev); vfio_unregister_group_dev(&vdev->vdev); vfio_pci_vf_uninit(vdev); - vfio_uninit_group_dev(&vdev->vdev); vfio_pci_vga_uninit(vdev); vfio_iommu_group_put(pdev->dev.iommu_group, &pdev->dev); if (!disable_idle_d3) vfio_pci_set_power_state(vdev, PCI_D0); - - mutex_destroy(&vdev->ioeventfds_lock); - kfree(vdev->region); - kfree(vdev->pm_save); - kfree(vdev); } static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, @@ -2018,16 +1942,13 @@ static pci_ers_result_t vfio_pci_aer_err_detected(struct pci_dev *pdev, return PCI_ERS_RESULT_CAN_RECOVER; } -static int vfio_pci_sriov_configure(struct pci_dev *pdev, int nr_virtfn) +int vfio_pci_core_sriov_configure(struct pci_dev *pdev, int nr_virtfn) { struct vfio_device *device; int ret = 0; might_sleep(); - if (!enable_sriov) - return -ENOENT; - device = vfio_device_get_from_dev(&pdev->dev); if (!device) return -ENODEV; @@ -2042,19 +1963,10 @@ static int vfio_pci_sriov_configure(struct pci_dev *pdev, int nr_virtfn) return ret < 0 ? ret : nr_virtfn; } -static const struct pci_error_handlers vfio_err_handlers = { +const struct pci_error_handlers vfio_pci_core_err_handlers = { .error_detected = vfio_pci_aer_err_detected, }; -static struct pci_driver vfio_pci_driver = { - .name = "vfio-pci", - .id_table = NULL, /* only dynamic ids */ - .probe = vfio_pci_probe, - .remove = vfio_pci_remove, - .sriov_configure = vfio_pci_sriov_configure, - .err_handler = &vfio_err_handlers, -}; - static int vfio_pci_check_all_devices_bound(struct pci_dev *pdev, void *data) { struct vfio_device_set *dev_set = data; @@ -2216,83 +2128,15 @@ static void vfio_pci_try_bus_reset(struct vfio_pci_core_device *vdev) } } -static void __exit vfio_pci_cleanup(void) +/* This will become the __exit function of vfio_pci_core.ko */ +void vfio_pci_core_cleanup(void) { - pci_unregister_driver(&vfio_pci_driver); vfio_pci_uninit_perm_bits(); } -static void __init vfio_pci_fill_ids(void) -{ - char *p, *id; - int rc; - - /* no ids passed actually */ - if (ids[0] == '\0') - return; - - /* add ids specified in the module parameter */ - p = ids; - while ((id = strsep(&p, ","))) { - unsigned int vendor, device, subvendor = PCI_ANY_ID, - subdevice = PCI_ANY_ID, class = 0, class_mask = 0; - int fields; - - if (!strlen(id)) - continue; - - fields = sscanf(id, "%x:%x:%x:%x:%x:%x", - &vendor, &device, &subvendor, &subdevice, - &class, &class_mask); - - if (fields < 2) { - pr_warn("invalid id string \"%s\"\n", id); - continue; - } - - rc = pci_add_dynid(&vfio_pci_driver, vendor, device, - subvendor, subdevice, class, class_mask, 0); - if (rc) - pr_warn("failed to add dynamic id [%04x:%04x[%04x:%04x]] class %#08x/%08x (%d)\n", - vendor, device, subvendor, subdevice, - class, class_mask, rc); - else - pr_info("add [%04x:%04x[%04x:%04x]] class %#08x/%08x\n", - vendor, device, subvendor, subdevice, - class, class_mask); - } -} - -static int __init vfio_pci_init(void) +/* This will become the __init function of vfio_pci_core.ko */ +int __init vfio_pci_core_init(void) { - int ret; - /* Allocate shared config space permission data used by all devices */ - ret = vfio_pci_init_perm_bits(); - if (ret) - return ret; - - /* Register and scan for devices */ - ret = pci_register_driver(&vfio_pci_driver); - if (ret) - goto out_driver; - - vfio_pci_fill_ids(); - - if (disable_denylist) - pr_warn("device denylist disabled.\n"); - - return 0; - -out_driver: - vfio_pci_uninit_perm_bits(); - return ret; + return vfio_pci_init_perm_bits(); } - -module_init(vfio_pci_init); -module_exit(vfio_pci_cleanup); - -MODULE_VERSION(DRIVER_VERSION); -MODULE_LICENSE("GPL v2"); -MODULE_AUTHOR(DRIVER_AUTHOR); -MODULE_DESCRIPTION(DRIVER_DESC); diff --git a/drivers/vfio/pci/vfio_pci_core.h b/drivers/vfio/pci/vfio_pci_core.h index 17ad048752b6..7dbdd4dda5c0 100644 --- a/drivers/vfio/pci/vfio_pci_core.h +++ b/drivers/vfio/pci/vfio_pci_core.h @@ -206,4 +206,27 @@ static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev, } #endif +/* Will be exported for vfio pci drivers usage */ +void vfio_pci_core_cleanup(void); +int vfio_pci_core_init(void); +void vfio_pci_core_close_device(struct vfio_device *core_vdev); +int vfio_pci_core_open_device(struct vfio_device *core_vdev); +void vfio_pci_core_init_device(struct vfio_pci_core_device *vdev, + struct pci_dev *pdev, + const struct vfio_device_ops *vfio_pci_ops); +int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev); +void vfio_pci_core_uninit_device(struct vfio_pci_core_device *vdev); +void vfio_pci_core_unregister_device(struct vfio_pci_core_device *vdev); +int vfio_pci_core_sriov_configure(struct pci_dev *pdev, int nr_virtfn); +extern const struct pci_error_handlers vfio_pci_core_err_handlers; +long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, + unsigned long arg); +ssize_t vfio_pci_core_read(struct vfio_device *core_vdev, char __user *buf, + size_t count, loff_t *ppos); +ssize_t vfio_pci_core_write(struct vfio_device *core_vdev, const char __user *buf, + size_t count, loff_t *ppos); +int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma); +void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count); +int vfio_pci_core_match(struct vfio_device *core_vdev, char *buf); + #endif /* VFIO_PCI_CORE_H */