From patchwork Mon Oct 5 17:36:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA1DC6CA for ; Mon, 5 Oct 2020 17:37:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A56EE20853 for ; Mon, 5 Oct 2020 17:37:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729034AbgJERhf (ORCPT ); Mon, 5 Oct 2020 13:37:35 -0400 Received: from inva021.nxp.com ([92.121.34.21]:44610 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728473AbgJERhB (ORCPT ); Mon, 5 Oct 2020 13:37:01 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 2C54E201586; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 1F5AE2007B9; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id C4B112032B; Mon, 5 Oct 2020 19:36:57 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Bharat Bhushan , Diana Craciun Subject: [PATCH v6 01/10] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices Date: Mon, 5 Oct 2020 20:36:45 +0300 Message-Id: <20201005173654.31773-2-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Bharat Bhushan DPAA2 (Data Path Acceleration Architecture) consists in mechanisms for processing Ethernet packets, queue management, accelerators, etc. The Management Complex (mc) is a hardware entity that manages the DPAA2 hardware resources. It provides an object-based abstraction for software drivers to use the DPAA2 hardware. The MC mediates operations such as create, discover, destroy of DPAA2 objects. The MC provides memory-mapped I/O command interfaces (MC portals) which DPAA2 software drivers use to operate on DPAA2 objects. A DPRC is a container object that holds other types of DPAA2 objects. Each object in the DPRC is a Linux device and bound to a driver. The MC-bus driver is a platform driver (different from PCI or platform bus). The DPRC driver does runtime management of a bus instance. It performs the initial scan of the DPRC and handles changes in the DPRC configuration (adding/removing objects). All objects inside a container share the same hardware isolation context, meaning that only an entire DPRC can be assigned to a virtual machine. When a container is assigned to a virtual machine, all the objects within that container are assigned to that virtual machine. The DPRC container assigned to the virtual machine is not allowed to change contents (add/remove objects) by the guest. The restriction is set by the host and enforced by the mc hardware. The DPAA2 objects can be directly assigned to the guest. However the MC portals (the memory mapped command interface to the MC) need to be emulated because there are commands that configure the interrupts and the isolation IDs which are virtual in the guest. Example: echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind The dprc.2 is bound to the VFIO driver and all the objects within dprc.2 are going to be bound to the VFIO driver. This patch adds the infrastructure for VFIO support for fsl-mc devices. Subsequent patches will add support for binding and secure assigning these devices using VFIO. More details about the DPAA2 objects can be found here: Documentation/networking/device_drivers/freescale/dpaa2/overview.rst Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- MAINTAINERS | 6 + drivers/vfio/Kconfig | 1 + drivers/vfio/Makefile | 1 + drivers/vfio/fsl-mc/Kconfig | 9 ++ drivers/vfio/fsl-mc/Makefile | 4 + drivers/vfio/fsl-mc/vfio_fsl_mc.c | 157 ++++++++++++++++++++++ drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 14 ++ include/uapi/linux/vfio.h | 1 + 8 files changed, 193 insertions(+) create mode 100644 drivers/vfio/fsl-mc/Kconfig create mode 100644 drivers/vfio/fsl-mc/Makefile create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h diff --git a/MAINTAINERS b/MAINTAINERS index 33b27e62ce19..1046f4065ac1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -18258,6 +18258,12 @@ F: drivers/vfio/ F: include/linux/vfio.h F: include/uapi/linux/vfio.h +VFIO FSL-MC DRIVER +M: Diana Craciun +L: kvm@vger.kernel.org +S: Maintained +F: drivers/vfio/fsl-mc/ + VFIO MEDIATED DEVICE DRIVERS M: Kirti Wankhede L: kvm@vger.kernel.org diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig index fd17db9b432f..5533df91b257 100644 --- a/drivers/vfio/Kconfig +++ b/drivers/vfio/Kconfig @@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU source "drivers/vfio/pci/Kconfig" source "drivers/vfio/platform/Kconfig" source "drivers/vfio/mdev/Kconfig" +source "drivers/vfio/fsl-mc/Kconfig" source "virt/lib/Kconfig" diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile index de67c4725cce..fee73f3d9480 100644 --- a/drivers/vfio/Makefile +++ b/drivers/vfio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o obj-$(CONFIG_VFIO_PCI) += pci/ obj-$(CONFIG_VFIO_PLATFORM) += platform/ obj-$(CONFIG_VFIO_MDEV) += mdev/ +obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/ diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig new file mode 100644 index 000000000000..b1a527d6b6f2 --- /dev/null +++ b/drivers/vfio/fsl-mc/Kconfig @@ -0,0 +1,9 @@ +config VFIO_FSL_MC + tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices" + depends on VFIO && FSL_MC_BUS && EVENTFD + help + Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc + (Management Complex) devices. This is required to passthrough + fsl-mc bus devices using the VFIO framework. + + If you don't know what to do here, say N. diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile new file mode 100644 index 000000000000..0c6e5d2ddaae --- /dev/null +++ b/drivers/vfio/fsl-mc/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) + +vfio-fsl-mc-y := vfio_fsl_mc.o +obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c new file mode 100644 index 000000000000..a7a483a1e90b --- /dev/null +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) +/* + * Copyright 2013-2016 Freescale Semiconductor Inc. + * Copyright 2016-2017,2019-2020 NXP + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vfio_fsl_mc_private.h" + +static int vfio_fsl_mc_open(void *device_data) +{ + if (!try_module_get(THIS_MODULE)) + return -ENODEV; + + return 0; +} + +static void vfio_fsl_mc_release(void *device_data) +{ + module_put(THIS_MODULE); +} + +static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, + unsigned long arg) +{ + switch (cmd) { + case VFIO_DEVICE_GET_INFO: + { + return -ENOTTY; + } + case VFIO_DEVICE_GET_REGION_INFO: + { + return -ENOTTY; + } + case VFIO_DEVICE_GET_IRQ_INFO: + { + return -ENOTTY; + } + case VFIO_DEVICE_SET_IRQS: + { + return -ENOTTY; + } + case VFIO_DEVICE_RESET: + { + return -ENOTTY; + } + default: + return -ENOTTY; + } +} + +static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf, + size_t count, loff_t *ppos) +{ + return -EINVAL; +} + +static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf, + size_t count, loff_t *ppos) +{ + return -EINVAL; +} + +static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma) +{ + return -EINVAL; +} + +static const struct vfio_device_ops vfio_fsl_mc_ops = { + .name = "vfio-fsl-mc", + .open = vfio_fsl_mc_open, + .release = vfio_fsl_mc_release, + .ioctl = vfio_fsl_mc_ioctl, + .read = vfio_fsl_mc_read, + .write = vfio_fsl_mc_write, + .mmap = vfio_fsl_mc_mmap, +}; + +static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) +{ + struct iommu_group *group; + struct vfio_fsl_mc_device *vdev; + struct device *dev = &mc_dev->dev; + int ret; + + group = vfio_iommu_group_get(dev); + if (!group) { + dev_err(dev, "VFIO_FSL_MC: No IOMMU group\n"); + return -EINVAL; + } + + vdev = devm_kzalloc(dev, sizeof(*vdev), GFP_KERNEL); + if (!vdev) { + ret = -ENOMEM; + goto out_group_put; + } + + vdev->mc_dev = mc_dev; + + ret = vfio_add_group_dev(dev, &vfio_fsl_mc_ops, vdev); + if (ret) { + dev_err(dev, "VFIO_FSL_MC: Failed to add to vfio group\n"); + goto out_group_put; + } + return 0; + +out_group_put: + vfio_iommu_group_put(group, dev); + return ret; +} + +static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev) +{ + struct vfio_fsl_mc_device *vdev; + struct device *dev = &mc_dev->dev; + + vdev = vfio_del_group_dev(dev); + if (!vdev) + return -EINVAL; + + vfio_iommu_group_put(mc_dev->dev.iommu_group, dev); + + return 0; +} + +static struct fsl_mc_driver vfio_fsl_mc_driver = { + .probe = vfio_fsl_mc_probe, + .remove = vfio_fsl_mc_remove, + .driver = { + .name = "vfio-fsl-mc", + .owner = THIS_MODULE, + }, +}; + +static int __init vfio_fsl_mc_driver_init(void) +{ + return fsl_mc_driver_register(&vfio_fsl_mc_driver); +} + +static void __exit vfio_fsl_mc_driver_exit(void) +{ + fsl_mc_driver_unregister(&vfio_fsl_mc_driver); +} + +module_init(vfio_fsl_mc_driver_init); +module_exit(vfio_fsl_mc_driver_exit); + +MODULE_LICENSE("Dual BSD/GPL"); +MODULE_DESCRIPTION("VFIO for FSL-MC devices - User Level meta-driver"); diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h new file mode 100644 index 000000000000..e79cc116f6b8 --- /dev/null +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) */ +/* + * Copyright 2013-2016 Freescale Semiconductor Inc. + * Copyright 2016,2019-2020 NXP + */ + +#ifndef VFIO_FSL_MC_PRIVATE_H +#define VFIO_FSL_MC_PRIVATE_H + +struct vfio_fsl_mc_device { + struct fsl_mc_device *mc_dev; +}; + +#endif /* VFIO_FSL_MC_PRIVATE_H */ diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 920470502329..95deac891378 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -201,6 +201,7 @@ struct vfio_device_info { #define VFIO_DEVICE_FLAGS_AMBA (1 << 3) /* vfio-amba device */ #define VFIO_DEVICE_FLAGS_CCW (1 << 4) /* vfio-ccw device */ #define VFIO_DEVICE_FLAGS_AP (1 << 5) /* vfio-ap device */ +#define VFIO_DEVICE_FLAGS_FSL_MC (1 << 6) /* vfio-fsl-mc device */ __u32 num_regions; /* Max region index + 1 */ __u32 num_irqs; /* Max IRQ index + 1 */ }; From patchwork Mon Oct 5 17:36:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817089 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A55AC618 for ; Mon, 5 Oct 2020 17:37:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9608C2080A for ; Mon, 5 Oct 2020 17:37:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728664AbgJERhD (ORCPT ); Mon, 5 Oct 2020 13:37:03 -0400 Received: from inva020.nxp.com ([92.121.34.13]:52078 "EHLO inva020.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728476AbgJERhA (ORCPT ); Mon, 5 Oct 2020 13:37:00 -0400 Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 8A5AD1A094A; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 7C5691A0940; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 2E1522032B; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind Date: Mon, 5 Oct 2020 20:36:46 +0300 Message-Id: <20201005173654.31773-3-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The DPRC (Data Path Resource Container) device is a bus device and has child devices attached to it. When the vfio-fsl-mc driver is probed the DPRC is scanned and the child devices discovered and initialized. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 91 +++++++++++++++++++++++ drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 1 + 2 files changed, 92 insertions(+) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index a7a483a1e90b..594760203268 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -15,6 +15,8 @@ #include "vfio_fsl_mc_private.h" +static struct fsl_mc_driver vfio_fsl_mc_driver; + static int vfio_fsl_mc_open(void *device_data) { if (!try_module_get(THIS_MODULE)) @@ -84,6 +86,80 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = { .mmap = vfio_fsl_mc_mmap, }; +static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb, + unsigned long action, void *data) +{ + struct vfio_fsl_mc_device *vdev = container_of(nb, + struct vfio_fsl_mc_device, nb); + struct device *dev = data; + struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev); + struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent); + + if (action == BUS_NOTIFY_ADD_DEVICE && + vdev->mc_dev == mc_cont) { + mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s", + vfio_fsl_mc_ops.name); + if (!mc_dev->driver_override) + dev_warn(dev, "VFIO_FSL_MC: Setting driver override for device in dprc %s failed\n", + dev_name(&mc_cont->dev)); + else + dev_info(dev, "VFIO_FSL_MC: Setting driver override for device in dprc %s\n", + dev_name(&mc_cont->dev)); + } else if (action == BUS_NOTIFY_BOUND_DRIVER && + vdev->mc_dev == mc_cont) { + struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver); + + if (mc_drv && mc_drv != &vfio_fsl_mc_driver) + dev_warn(dev, "VFIO_FSL_MC: Object %s bound to driver %s while DPRC bound to vfio-fsl-mc\n", + dev_name(dev), mc_drv->driver.name); + } + + return 0; +} + +static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev) +{ + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int ret; + + /* Non-dprc devices share mc_io from parent */ + if (!is_fsl_mc_bus_dprc(mc_dev)) { + struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent); + + mc_dev->mc_io = mc_cont->mc_io; + return 0; + } + + vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier; + ret = bus_register_notifier(&fsl_mc_bus_type, &vdev->nb); + if (ret) + return ret; + + /* open DPRC, allocate a MC portal */ + ret = dprc_setup(mc_dev); + if (ret) { + dev_err(&mc_dev->dev, "VFIO_FSL_MC: Failed to setup DPRC (%d)\n", ret); + goto out_nc_unreg; + } + + ret = dprc_scan_container(mc_dev, false); + if (ret) { + dev_err(&mc_dev->dev, "VFIO_FSL_MC: Container scanning failed (%d)\n", ret); + goto out_dprc_cleanup; + } + + return 0; + +out_dprc_cleanup: + dprc_remove_devices(mc_dev, NULL, 0); + dprc_cleanup(mc_dev); +out_nc_unreg: + bus_unregister_notifier(&fsl_mc_bus_type, &vdev->nb); + vdev->nb.notifier_call = NULL; + + return ret; +} + static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) { struct iommu_group *group; @@ -110,8 +186,15 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) dev_err(dev, "VFIO_FSL_MC: Failed to add to vfio group\n"); goto out_group_put; } + + ret = vfio_fsl_mc_init_device(vdev); + if (ret) + goto out_group_dev; + return 0; +out_group_dev: + vfio_del_group_dev(dev); out_group_put: vfio_iommu_group_put(group, dev); return ret; @@ -126,6 +209,14 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev) if (!vdev) return -EINVAL; + if (is_fsl_mc_bus_dprc(mc_dev)) { + dprc_remove_devices(mc_dev, NULL, 0); + dprc_cleanup(mc_dev); + } + + if (vdev->nb.notifier_call) + bus_unregister_notifier(&fsl_mc_bus_type, &vdev->nb); + vfio_iommu_group_put(mc_dev->dev.iommu_group, dev); return 0; diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index e79cc116f6b8..37d61eaa58c8 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -9,6 +9,7 @@ struct vfio_fsl_mc_device { struct fsl_mc_device *mc_dev; + struct notifier_block nb; }; #endif /* VFIO_FSL_MC_PRIVATE_H */ From patchwork Mon Oct 5 17:36:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817087 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C17F6CA for ; Mon, 5 Oct 2020 17:37:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B6712083B for ; Mon, 5 Oct 2020 17:37:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728588AbgJERhA (ORCPT ); Mon, 5 Oct 2020 13:37:00 -0400 Received: from inva020.nxp.com ([92.121.34.13]:52106 "EHLO inva020.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728479AbgJERhA (ORCPT ); Mon, 5 Oct 2020 13:37:00 -0400 Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id EA8A11A0953; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id DC95B1A091E; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 8B0852032B; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 03/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl Date: Mon, 5 Oct 2020 20:36:47 +0300 Message-Id: <20201005173654.31773-4-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allow userspace to get fsl-mc device info (number of regions and irqs). Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 594760203268..161c2cbe07dc 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -33,10 +33,29 @@ static void vfio_fsl_mc_release(void *device_data) static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, unsigned long arg) { + unsigned long minsz; + struct vfio_fsl_mc_device *vdev = device_data; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + switch (cmd) { case VFIO_DEVICE_GET_INFO: { - return -ENOTTY; + struct vfio_device_info info; + + minsz = offsetofend(struct vfio_device_info, num_irqs); + + if (copy_from_user(&info, (void __user *)arg, minsz)) + return -EFAULT; + + if (info.argsz < minsz) + return -EINVAL; + + info.flags = VFIO_DEVICE_FLAGS_FSL_MC; + info.num_regions = mc_dev->obj_desc.region_count; + info.num_irqs = mc_dev->obj_desc.irq_count; + + return copy_to_user((void __user *)arg, &info, minsz) ? + -EFAULT : 0; } case VFIO_DEVICE_GET_REGION_INFO: { From patchwork Mon Oct 5 17:36:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817103 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 10A59618 for ; Mon, 5 Oct 2020 17:37:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 026C82080A for ; Mon, 5 Oct 2020 17:37:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729042AbgJERhg (ORCPT ); Mon, 5 Oct 2020 13:37:36 -0400 Received: from inva021.nxp.com ([92.121.34.21]:44668 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728492AbgJERhB (ORCPT ); Mon, 5 Oct 2020 13:37:01 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 4CD762015AD; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 49C822007BB; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id EAFD12032B; Mon, 5 Oct 2020 19:36:58 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 04/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call Date: Mon, 5 Oct 2020 20:36:48 +0300 Message-Id: <20201005173654.31773-5-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Expose to userspace information about the memory regions. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 79 ++++++++++++++++++++++- drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 18 ++++++ 2 files changed, 96 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 161c2cbe07dc..05dace5ddc2c 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -17,16 +17,71 @@ static struct fsl_mc_driver vfio_fsl_mc_driver; +static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev) +{ + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int count = mc_dev->obj_desc.region_count; + int i; + + vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region), + GFP_KERNEL); + if (!vdev->regions) + return -ENOMEM; + + for (i = 0; i < count; i++) { + struct resource *res = &mc_dev->regions[i]; + + vdev->regions[i].addr = res->start; + vdev->regions[i].size = resource_size(res); + vdev->regions[i].flags = 0; + vdev->regions[i].type = mc_dev->regions[i].flags & IORESOURCE_BITS; + } + + return 0; +} + +static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev) +{ + kfree(vdev->regions); +} + static int vfio_fsl_mc_open(void *device_data) { + struct vfio_fsl_mc_device *vdev = device_data; + int ret; + if (!try_module_get(THIS_MODULE)) return -ENODEV; + mutex_lock(&vdev->driver_lock); + if (!vdev->refcnt) { + ret = vfio_fsl_mc_regions_init(vdev); + if (ret) + goto err_reg_init; + } + vdev->refcnt++; + + mutex_unlock(&vdev->driver_lock); + return 0; + +err_reg_init: + mutex_unlock(&vdev->driver_lock); + module_put(THIS_MODULE); + return ret; } static void vfio_fsl_mc_release(void *device_data) { + struct vfio_fsl_mc_device *vdev = device_data; + + mutex_lock(&vdev->driver_lock); + + if (!(--vdev->refcnt)) + vfio_fsl_mc_regions_cleanup(vdev); + + mutex_unlock(&vdev->driver_lock); + module_put(THIS_MODULE); } @@ -59,7 +114,25 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, } case VFIO_DEVICE_GET_REGION_INFO: { - return -ENOTTY; + struct vfio_region_info info; + + minsz = offsetofend(struct vfio_region_info, offset); + + if (copy_from_user(&info, (void __user *)arg, minsz)) + return -EFAULT; + + if (info.argsz < minsz) + return -EINVAL; + + if (info.index >= mc_dev->obj_desc.region_count) + return -EINVAL; + + /* map offset to the physical address */ + info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index); + info.size = vdev->regions[info.index].size; + info.flags = vdev->regions[info.index].flags; + + return copy_to_user((void __user *)arg, &info, minsz); } case VFIO_DEVICE_GET_IRQ_INFO: { @@ -210,6 +283,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) if (ret) goto out_group_dev; + mutex_init(&vdev->driver_lock); + return 0; out_group_dev: @@ -228,6 +303,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev) if (!vdev) return -EINVAL; + mutex_destroy(&vdev->driver_lock); + if (is_fsl_mc_bus_dprc(mc_dev)) { dprc_remove_devices(mc_dev, NULL, 0); dprc_cleanup(mc_dev); diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index 37d61eaa58c8..be60f41af30f 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -7,9 +7,27 @@ #ifndef VFIO_FSL_MC_PRIVATE_H #define VFIO_FSL_MC_PRIVATE_H +#define VFIO_FSL_MC_OFFSET_SHIFT 40 +#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1) + +#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT) + +#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \ + ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT) + +struct vfio_fsl_mc_region { + u32 flags; + u32 type; + u64 addr; + resource_size_t size; +}; + struct vfio_fsl_mc_device { struct fsl_mc_device *mc_dev; struct notifier_block nb; + int refcnt; + struct vfio_fsl_mc_region *regions; + struct mutex driver_lock; }; #endif /* VFIO_FSL_MC_PRIVATE_H */ From patchwork Mon Oct 5 17:36:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817099 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B06B112E for ; Mon, 5 Oct 2020 17:37:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2CEF720853 for ; Mon, 5 Oct 2020 17:37:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728991AbgJERhW (ORCPT ); Mon, 5 Oct 2020 13:37:22 -0400 Received: from inva021.nxp.com ([92.121.34.21]:44692 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727344AbgJERhB (ORCPT ); Mon, 5 Oct 2020 13:37:01 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id B416A20159F; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id A76C62015AB; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 58A942032B; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 05/10] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions Date: Mon, 5 Oct 2020 20:36:49 +0300 Message-Id: <20201005173654.31773-6-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allow userspace to mmap device regions for direct access of fsl-mc devices. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 68 ++++++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 05dace5ddc2c..55190a2730fb 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -30,11 +30,20 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev) for (i = 0; i < count; i++) { struct resource *res = &mc_dev->regions[i]; + int no_mmap = is_fsl_mc_bus_dprc(mc_dev); vdev->regions[i].addr = res->start; vdev->regions[i].size = resource_size(res); - vdev->regions[i].flags = 0; vdev->regions[i].type = mc_dev->regions[i].flags & IORESOURCE_BITS; + /* + * Only regions addressed with PAGE granularity may be + * MMAPed securely. + */ + if (!no_mmap && !(vdev->regions[i].addr & ~PAGE_MASK) && + !(vdev->regions[i].size & ~PAGE_MASK)) + vdev->regions[i].flags |= + VFIO_REGION_INFO_FLAG_MMAP; + } return 0; @@ -163,9 +172,64 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf, return -EINVAL; } +static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region, + struct vm_area_struct *vma) +{ + u64 size = vma->vm_end - vma->vm_start; + u64 pgoff, base; + u8 region_cacheable; + + pgoff = vma->vm_pgoff & + ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1); + base = pgoff << PAGE_SHIFT; + + if (region.size < PAGE_SIZE || base + size > region.size) + return -EINVAL; + + region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) && + (region.type & FSL_MC_REGION_SHAREABLE); + if (!region_cacheable) + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + + vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff; + + return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, + size, vma->vm_page_prot); +} + static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma) { - return -EINVAL; + struct vfio_fsl_mc_device *vdev = device_data; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int index; + + index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT); + + if (vma->vm_end < vma->vm_start) + return -EINVAL; + if (vma->vm_start & ~PAGE_MASK) + return -EINVAL; + if (vma->vm_end & ~PAGE_MASK) + return -EINVAL; + if (!(vma->vm_flags & VM_SHARED)) + return -EINVAL; + if (index >= mc_dev->obj_desc.region_count) + return -EINVAL; + + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP)) + return -EINVAL; + + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ) + && (vma->vm_flags & VM_READ)) + return -EINVAL; + + if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE) + && (vma->vm_flags & VM_WRITE)) + return -EINVAL; + + vma->vm_private_data = mc_dev; + + return vfio_fsl_mc_mmap_mmio(vdev->regions[index], vma); } static const struct vfio_device_ops vfio_fsl_mc_ops = { From patchwork Mon Oct 5 17:36:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817101 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 861936CA for ; Mon, 5 Oct 2020 17:37:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 78A652080A for ; Mon, 5 Oct 2020 17:37:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728933AbgJERhV (ORCPT ); Mon, 5 Oct 2020 13:37:21 -0400 Received: from inva020.nxp.com ([92.121.34.13]:52148 "EHLO inva020.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728565AbgJERhC (ORCPT ); Mon, 5 Oct 2020 13:37:02 -0400 Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 184191A0943; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 08FC31A0940; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id B5FBA2032B; Mon, 5 Oct 2020 19:36:59 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun Subject: [PATCH v6 06/10] vfio/fsl-mc: Added lock support in preparation for interrupt handling Date: Mon, 5 Oct 2020 20:36:50 +0300 Message-Id: <20201005173654.31773-7-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Only the DPRC object allocates interrupts from the MSI interrupt domain. The interrupts are managed by the DPRC in a pool of interrupts. The access to this pool of interrupts has to be protected with a lock. This patch extends the current lock implementation to have a lock per DPRC. Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 92 +++++++++++++++++++++-- drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 7 +- 2 files changed, 90 insertions(+), 9 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 55190a2730fb..b52407c4e1ea 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -17,6 +17,78 @@ static struct fsl_mc_driver vfio_fsl_mc_driver; +static DEFINE_MUTEX(reflck_lock); + +static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck) +{ + kref_get(&reflck->kref); +} + +static void vfio_fsl_mc_reflck_release(struct kref *kref) +{ + struct vfio_fsl_mc_reflck *reflck = container_of(kref, + struct vfio_fsl_mc_reflck, + kref); + + mutex_destroy(&reflck->lock); + kfree(reflck); + mutex_unlock(&reflck_lock); +} + +static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck) +{ + kref_put_mutex(&reflck->kref, vfio_fsl_mc_reflck_release, &reflck_lock); +} + +static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void) +{ + struct vfio_fsl_mc_reflck *reflck; + + reflck = kzalloc(sizeof(*reflck), GFP_KERNEL); + if (!reflck) + return ERR_PTR(-ENOMEM); + + kref_init(&reflck->kref); + mutex_init(&reflck->lock); + + return reflck; +} + +static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev) +{ + int ret; + + mutex_lock(&reflck_lock); + if (is_fsl_mc_bus_dprc(vdev->mc_dev)) { + vdev->reflck = vfio_fsl_mc_reflck_alloc(); + ret = PTR_ERR_OR_ZERO(vdev->reflck); + } else { + struct device *mc_cont_dev = vdev->mc_dev->dev.parent; + struct vfio_device *device; + struct vfio_fsl_mc_device *cont_vdev; + + device = vfio_device_get_from_dev(mc_cont_dev); + if (!device) { + ret = -ENODEV; + goto unlock; + } + + cont_vdev = vfio_device_data(device); + if (!cont_vdev || !cont_vdev->reflck) { + vfio_device_put(device); + ret = -ENODEV; + goto unlock; + } + vfio_fsl_mc_reflck_get(cont_vdev->reflck); + vdev->reflck = cont_vdev->reflck; + vfio_device_put(device); + } + +unlock: + mutex_unlock(&reflck_lock); + return ret; +} + static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev) { struct fsl_mc_device *mc_dev = vdev->mc_dev; @@ -62,7 +134,7 @@ static int vfio_fsl_mc_open(void *device_data) if (!try_module_get(THIS_MODULE)) return -ENODEV; - mutex_lock(&vdev->driver_lock); + mutex_lock(&vdev->reflck->lock); if (!vdev->refcnt) { ret = vfio_fsl_mc_regions_init(vdev); if (ret) @@ -70,12 +142,12 @@ static int vfio_fsl_mc_open(void *device_data) } vdev->refcnt++; - mutex_unlock(&vdev->driver_lock); + mutex_unlock(&vdev->reflck->lock); return 0; err_reg_init: - mutex_unlock(&vdev->driver_lock); + mutex_unlock(&vdev->reflck->lock); module_put(THIS_MODULE); return ret; } @@ -84,12 +156,12 @@ static void vfio_fsl_mc_release(void *device_data) { struct vfio_fsl_mc_device *vdev = device_data; - mutex_lock(&vdev->driver_lock); + mutex_lock(&vdev->reflck->lock); if (!(--vdev->refcnt)) vfio_fsl_mc_regions_cleanup(vdev); - mutex_unlock(&vdev->driver_lock); + mutex_unlock(&vdev->reflck->lock); module_put(THIS_MODULE); } @@ -343,14 +415,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) goto out_group_put; } - ret = vfio_fsl_mc_init_device(vdev); + ret = vfio_fsl_mc_reflck_attach(vdev); if (ret) goto out_group_dev; - mutex_init(&vdev->driver_lock); + ret = vfio_fsl_mc_init_device(vdev); + if (ret) + goto out_reflck; return 0; +out_reflck: + vfio_fsl_mc_reflck_put(vdev->reflck); out_group_dev: vfio_del_group_dev(dev); out_group_put: @@ -367,7 +443,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev) if (!vdev) return -EINVAL; - mutex_destroy(&vdev->driver_lock); + vfio_fsl_mc_reflck_put(vdev->reflck); if (is_fsl_mc_bus_dprc(mc_dev)) { dprc_remove_devices(mc_dev, NULL, 0); diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index be60f41af30f..d47ef6215429 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -15,6 +15,11 @@ #define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \ ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT) +struct vfio_fsl_mc_reflck { + struct kref kref; + struct mutex lock; +}; + struct vfio_fsl_mc_region { u32 flags; u32 type; @@ -27,7 +32,7 @@ struct vfio_fsl_mc_device { struct notifier_block nb; int refcnt; struct vfio_fsl_mc_region *regions; - struct mutex driver_lock; + struct vfio_fsl_mc_reflck *reflck; }; #endif /* VFIO_FSL_MC_PRIVATE_H */ From patchwork Mon Oct 5 17:36:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817097 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DBE326CA for ; Mon, 5 Oct 2020 17:37:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C981920853 for ; Mon, 5 Oct 2020 17:37:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728970AbgJERhV (ORCPT ); Mon, 5 Oct 2020 13:37:21 -0400 Received: from inva020.nxp.com ([92.121.34.13]:52162 "EHLO inva020.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728604AbgJERhC (ORCPT ); Mon, 5 Oct 2020 13:37:02 -0400 Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 7B00B1A0940; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 6B8CC1A091E; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 18B662032B; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 07/10] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices Date: Mon, 5 Oct 2020 20:36:51 +0300 Message-Id: <20201005173654.31773-8-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch adds the skeleton for interrupt support for fsl-mc devices. The interrupts are not yet functional, the functionality will be added by subsequent patches. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/Makefile | 2 +- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 52 ++++++++++++++++++++++- drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 34 +++++++++++++++ drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 6 +++ 4 files changed, 91 insertions(+), 3 deletions(-) create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile index 0c6e5d2ddaae..cad6dbf0b735 100644 --- a/drivers/vfio/fsl-mc/Makefile +++ b/drivers/vfio/fsl-mc/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) -vfio-fsl-mc-y := vfio_fsl_mc.o +vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index b52407c4e1ea..7803a9d6bfd9 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -217,11 +217,55 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, } case VFIO_DEVICE_GET_IRQ_INFO: { - return -ENOTTY; + struct vfio_irq_info info; + + minsz = offsetofend(struct vfio_irq_info, count); + if (copy_from_user(&info, (void __user *)arg, minsz)) + return -EFAULT; + + if (info.argsz < minsz) + return -EINVAL; + + if (info.index >= mc_dev->obj_desc.irq_count) + return -EINVAL; + + info.flags = VFIO_IRQ_INFO_EVENTFD; + info.count = 1; + + return copy_to_user((void __user *)arg, &info, minsz); } case VFIO_DEVICE_SET_IRQS: { - return -ENOTTY; + struct vfio_irq_set hdr; + u8 *data = NULL; + int ret = 0; + size_t data_size = 0; + + minsz = offsetofend(struct vfio_irq_set, count); + + if (copy_from_user(&hdr, (void __user *)arg, minsz)) + return -EFAULT; + + ret = vfio_set_irqs_validate_and_prepare(&hdr, mc_dev->obj_desc.irq_count, + mc_dev->obj_desc.irq_count, &data_size); + if (ret) + return ret; + + if (data_size) { + data = memdup_user((void __user *)(arg + minsz), + data_size); + if (IS_ERR(data)) + return PTR_ERR(data); + } + + mutex_lock(&vdev->igate); + ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags, + hdr.index, hdr.start, + hdr.count, data); + mutex_unlock(&vdev->igate); + kfree(data); + + return ret; } case VFIO_DEVICE_RESET: { @@ -423,6 +467,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev) if (ret) goto out_reflck; + mutex_init(&vdev->igate); + return 0; out_reflck: @@ -443,6 +489,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev) if (!vdev) return -EINVAL; + mutex_destroy(&vdev->igate); + vfio_fsl_mc_reflck_put(vdev->reflck); if (is_fsl_mc_bus_dprc(mc_dev)) { diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c new file mode 100644 index 000000000000..5232f208e361 --- /dev/null +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause) +/* + * Copyright 2013-2016 Freescale Semiconductor Inc. + * Copyright 2019 NXP + */ + +#include +#include +#include +#include +#include + +#include "linux/fsl/mc.h" +#include "vfio_fsl_mc_private.h" + +static int vfio_fsl_mc_set_irq_trigger(struct vfio_fsl_mc_device *vdev, + unsigned int index, unsigned int start, + unsigned int count, u32 flags, + void *data) +{ + return -EINVAL; +} + +int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, + u32 flags, unsigned int index, + unsigned int start, unsigned int count, + void *data) +{ + if (flags & VFIO_IRQ_SET_ACTION_TRIGGER) + return vfio_fsl_mc_set_irq_trigger(vdev, index, start, + count, flags, data); + else + return -EINVAL; +} diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index d47ef6215429..2c3f625a3240 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -33,6 +33,12 @@ struct vfio_fsl_mc_device { int refcnt; struct vfio_fsl_mc_region *regions; struct vfio_fsl_mc_reflck *reflck; + struct mutex igate; }; +extern int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, + u32 flags, unsigned int index, + unsigned int start, unsigned int count, + void *data); + #endif /* VFIO_FSL_MC_PRIVATE_H */ From patchwork Mon Oct 5 17:36:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817091 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C7F5618 for ; Mon, 5 Oct 2020 17:37:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 395C42083B for ; Mon, 5 Oct 2020 17:37:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728697AbgJERhH (ORCPT ); Mon, 5 Oct 2020 13:37:07 -0400 Received: from inva021.nxp.com ([92.121.34.21]:44722 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728634AbgJERhD (ORCPT ); Mon, 5 Oct 2020 13:37:03 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id D45A12007B2; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id C6F6F2007AE; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 77D3B2032B; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 08/10] vfio/fsl-mc: trigger an interrupt via eventfd Date: Mon, 5 Oct 2020 20:36:52 +0300 Message-Id: <20201005173654.31773-9-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch allows to set an eventfd for fsl-mc device interrupts and also to trigger the interrupt eventfd from userspace for testing. All fsl-mc device interrupts are MSIs. The MSIs are allocated from the MSI domain only once per DPRC and used by all the DPAA2 objects. The interrupts are managed by the DPRC in a pool of interrupts. Each device requests interrupts from this pool. The pool is allocated when the first virtual device is setting the interrupts. The pool of interrupts is protected by a lock. The DPRC has an interrupt of its own which indicates if the DPRC contents have changed. However, currently, the contents of a DPRC assigned to the guest cannot be changed at runtime, so this interrupt is not configured. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 24 +++- drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 162 +++++++++++++++++++++- drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 10 ++ 3 files changed, 194 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index 7803a9d6bfd9..feb5dafd4bf5 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -155,12 +155,34 @@ static int vfio_fsl_mc_open(void *device_data) static void vfio_fsl_mc_release(void *device_data) { struct vfio_fsl_mc_device *vdev = device_data; + int ret; mutex_lock(&vdev->reflck->lock); - if (!(--vdev->refcnt)) + if (!(--vdev->refcnt)) { + struct fsl_mc_device *mc_dev = vdev->mc_dev; + struct device *cont_dev = fsl_mc_cont_dev(&mc_dev->dev); + struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev); + vfio_fsl_mc_regions_cleanup(vdev); + /* reset the device before cleaning up the interrupts */ + ret = dprc_reset_container(mc_cont->mc_io, 0, + mc_cont->mc_handle, + mc_cont->obj_desc.id, + DPRC_RESET_OPTION_NON_RECURSIVE); + + if (ret) { + dev_warn(&mc_cont->dev, "VFIO_FLS_MC: reset device has failed (%d)\n", + ret); + WARN_ON(1); + } + + vfio_fsl_mc_irqs_cleanup(vdev); + + fsl_mc_cleanup_irq_pool(mc_cont); + } + mutex_unlock(&vdev->reflck->lock); module_put(THIS_MODULE); diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c index 5232f208e361..2ce2acad3461 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c @@ -13,12 +13,150 @@ #include "linux/fsl/mc.h" #include "vfio_fsl_mc_private.h" +int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev) +{ + struct fsl_mc_device *mc_dev = vdev->mc_dev; + struct vfio_fsl_mc_irq *mc_irq; + int irq_count; + int ret, i; + + /* Device does not support any interrupt */ + if (mc_dev->obj_desc.irq_count == 0) + return 0; + + /* interrupts were already allocated for this device */ + if (vdev->mc_irqs) + return 0; + + irq_count = mc_dev->obj_desc.irq_count; + + mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL); + if (!mc_irq) + return -ENOMEM; + + /* Allocate IRQs */ + ret = fsl_mc_allocate_irqs(mc_dev); + if (ret) { + kfree(mc_irq); + return ret; + } + + for (i = 0; i < irq_count; i++) { + mc_irq[i].count = 1; + mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD; + } + + vdev->mc_irqs = mc_irq; + + return 0; +} + +static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg) +{ + struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg; + + eventfd_signal(mc_irq->trigger, 1); + return IRQ_HANDLED; +} + +static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev, + int index, int fd) +{ + struct vfio_fsl_mc_irq *irq = &vdev->mc_irqs[index]; + struct eventfd_ctx *trigger; + int hwirq; + int ret; + + hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq; + if (irq->trigger) { + free_irq(hwirq, irq); + kfree(irq->name); + eventfd_ctx_put(irq->trigger); + irq->trigger = NULL; + } + + if (fd < 0) /* Disable only */ + return 0; + + irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)", + hwirq, dev_name(&vdev->mc_dev->dev)); + if (!irq->name) + return -ENOMEM; + + trigger = eventfd_ctx_fdget(fd); + if (IS_ERR(trigger)) { + kfree(irq->name); + return PTR_ERR(trigger); + } + + irq->trigger = trigger; + + ret = request_irq(hwirq, vfio_fsl_mc_irq_handler, 0, + irq->name, irq); + if (ret) { + kfree(irq->name); + eventfd_ctx_put(trigger); + irq->trigger = NULL; + return ret; + } + + return 0; +} + static int vfio_fsl_mc_set_irq_trigger(struct vfio_fsl_mc_device *vdev, unsigned int index, unsigned int start, unsigned int count, u32 flags, void *data) { - return -EINVAL; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int ret, hwirq; + struct vfio_fsl_mc_irq *irq; + struct device *cont_dev = fsl_mc_cont_dev(&mc_dev->dev); + struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev); + + if (start != 0 || count != 1) + return -EINVAL; + + mutex_lock(&vdev->reflck->lock); + ret = fsl_mc_populate_irq_pool(mc_cont, + FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS); + if (ret) + goto unlock; + + ret = vfio_fsl_mc_irqs_allocate(vdev); + if (ret) + goto unlock; + mutex_unlock(&vdev->reflck->lock); + + if (!count && (flags & VFIO_IRQ_SET_DATA_NONE)) + return vfio_set_trigger(vdev, index, -1); + + if (flags & VFIO_IRQ_SET_DATA_EVENTFD) { + s32 fd = *(s32 *)data; + + return vfio_set_trigger(vdev, index, fd); + } + + hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq; + + irq = &vdev->mc_irqs[index]; + + if (flags & VFIO_IRQ_SET_DATA_NONE) { + vfio_fsl_mc_irq_handler(hwirq, irq); + + } else if (flags & VFIO_IRQ_SET_DATA_BOOL) { + u8 trigger = *(u8 *)data; + + if (trigger) + vfio_fsl_mc_irq_handler(hwirq, irq); + } + + return 0; + +unlock: + mutex_unlock(&vdev->reflck->lock); + return ret; + } int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, @@ -32,3 +170,25 @@ int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, else return -EINVAL; } + +/* Free All IRQs for the given MC object */ +void vfio_fsl_mc_irqs_cleanup(struct vfio_fsl_mc_device *vdev) +{ + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int irq_count = mc_dev->obj_desc.irq_count; + int i; + + /* + * Device does not support any interrupt or the interrupts + * were not configured + */ + if (!vdev->mc_irqs) + return; + + for (i = 0; i < irq_count; i++) + vfio_set_trigger(vdev, i, -1); + + fsl_mc_free_irqs(mc_dev); + kfree(vdev->mc_irqs); + vdev->mc_irqs = NULL; +} diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index 2c3f625a3240..7aa49b9ba60d 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -15,6 +15,13 @@ #define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \ ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT) +struct vfio_fsl_mc_irq { + u32 flags; + u32 count; + struct eventfd_ctx *trigger; + char *name; +}; + struct vfio_fsl_mc_reflck { struct kref kref; struct mutex lock; @@ -34,6 +41,7 @@ struct vfio_fsl_mc_device { struct vfio_fsl_mc_region *regions; struct vfio_fsl_mc_reflck *reflck; struct mutex igate; + struct vfio_fsl_mc_irq *mc_irqs; }; extern int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, @@ -41,4 +49,6 @@ extern int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev, unsigned int start, unsigned int count, void *data); +void vfio_fsl_mc_irqs_cleanup(struct vfio_fsl_mc_device *vdev); + #endif /* VFIO_FSL_MC_PRIVATE_H */ From patchwork Mon Oct 5 17:36:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817093 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 272A06CA for ; Mon, 5 Oct 2020 17:37:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 147B32080A for ; Mon, 5 Oct 2020 17:37:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728634AbgJERhJ (ORCPT ); Mon, 5 Oct 2020 13:37:09 -0400 Received: from inva021.nxp.com ([92.121.34.21]:44738 "EHLO inva021.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728643AbgJERhD (ORCPT ); Mon, 5 Oct 2020 13:37:03 -0400 Received: from inva021.nxp.com (localhost [127.0.0.1]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 3D8AD2007B7; Mon, 5 Oct 2020 19:37:01 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva021.eu-rdc02.nxp.com (Postfix) with ESMTP id 317382007AE; Mon, 5 Oct 2020 19:37:01 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id D5CAE2032B; Mon, 5 Oct 2020 19:37:00 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun , Bharat Bhushan Subject: [PATCH v6 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices Date: Mon, 5 Oct 2020 20:36:53 +0300 Message-Id: <20201005173654.31773-10-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The software uses a memory-mapped I/O command interface (MC portals) to communicate with the MC hardware. This command interface is used to discover, enumerate, configure and remove DPAA2 objects. The DPAA2 objects use MSIs, so the command interface needs to be emulated such that the correct MSI is configured in the hardware (the guest has the virtual MSIs). This patch is adding read/write support for fsl-mc devices. The mc commands are emulated by the userspace. The host is just passing the correct command to the hardware. Also the current patch limits userspace to write complete 64byte command once and read 64byte response by one ioctl. Signed-off-by: Bharat Bhushan Signed-off-by: Diana Craciun --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 118 +++++++++++++++++++++- drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 1 + 2 files changed, 116 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index feb5dafd4bf5..d95568cd8021 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "vfio_fsl_mc_private.h" @@ -115,7 +116,9 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev) !(vdev->regions[i].size & ~PAGE_MASK)) vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_MMAP; - + vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ; + if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY)) + vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE; } return 0; @@ -123,6 +126,11 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev) static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev) { + struct fsl_mc_device *mc_dev = vdev->mc_dev; + int i; + + for (i = 0; i < mc_dev->obj_desc.region_count; i++) + iounmap(vdev->regions[i].ioaddr); kfree(vdev->regions); } @@ -301,13 +309,117 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf, size_t count, loff_t *ppos) { - return -EINVAL; + struct vfio_fsl_mc_device *vdev = device_data; + unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos); + loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + struct vfio_fsl_mc_region *region; + u64 data[8]; + int i; + + if (index >= mc_dev->obj_desc.region_count) + return -EINVAL; + + region = &vdev->regions[index]; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_READ)) + return -EINVAL; + + if (!region->ioaddr) { + region->ioaddr = ioremap(region->addr, region->size); + if (!region->ioaddr) + return -ENOMEM; + } + + if (count != 64 || off != 0) + return -EINVAL; + + for (i = 7; i >= 0; i--) + data[i] = readq(region->ioaddr + i * sizeof(uint64_t)); + + if (copy_to_user(buf, data, 64)) + return -EFAULT; + + return count; +} + +#define MC_CMD_COMPLETION_TIMEOUT_MS 5000 +#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS 500 + +static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data) +{ + int i; + enum mc_cmd_status status; + unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000; + + /* Write at command parameter into portal */ + for (i = 7; i >= 1; i--) + writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t)); + + /* Write command header in the end */ + writeq(cmd_data[0], ioaddr); + + /* Wait for response before returning to user-space + * This can be optimized in future to even prepare response + * before returning to user-space and avoid read ioctl. + */ + for (;;) { + u64 header; + struct mc_cmd_header *resp_hdr; + + header = cpu_to_le64(readq_relaxed(ioaddr)); + + resp_hdr = (struct mc_cmd_header *)&header; + status = (enum mc_cmd_status)resp_hdr->status; + if (status != MC_CMD_STATUS_READY) + break; + + udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS); + timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS; + if (timeout_usecs == 0) + return -ETIMEDOUT; + } + + return 0; } static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf, size_t count, loff_t *ppos) { - return -EINVAL; + struct vfio_fsl_mc_device *vdev = device_data; + unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos); + loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + struct vfio_fsl_mc_region *region; + u64 data[8]; + int ret; + + if (index >= mc_dev->obj_desc.region_count) + return -EINVAL; + + region = &vdev->regions[index]; + + if (!(region->flags & VFIO_REGION_INFO_FLAG_WRITE)) + return -EINVAL; + + if (!region->ioaddr) { + region->ioaddr = ioremap(region->addr, region->size); + if (!region->ioaddr) + return -ENOMEM; + } + + if (count != 64 || off != 0) + return -EINVAL; + + if (copy_from_user(&data, buf, 64)) + return -EFAULT; + + ret = vfio_fsl_mc_send_command(region->ioaddr, data); + if (ret) + return ret; + + return count; + } static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region, diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h index 7aa49b9ba60d..a97ee691ed47 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h @@ -32,6 +32,7 @@ struct vfio_fsl_mc_region { u32 type; u64 addr; resource_size_t size; + void __iomem *ioaddr; }; struct vfio_fsl_mc_device { From patchwork Mon Oct 5 17:36:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diana Madalina Craciun X-Patchwork-Id: 11817095 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC23B618 for ; Mon, 5 Oct 2020 17:37:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CDDB92080A for ; Mon, 5 Oct 2020 17:37:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728720AbgJERhI (ORCPT ); Mon, 5 Oct 2020 13:37:08 -0400 Received: from inva020.nxp.com ([92.121.34.13]:52188 "EHLO inva020.nxp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728479AbgJERhD (ORCPT ); Mon, 5 Oct 2020 13:37:03 -0400 Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 9343F1A0952; Mon, 5 Oct 2020 19:37:01 +0200 (CEST) Received: from inva024.eu-rdc02.nxp.com (inva024.eu-rdc02.nxp.com [134.27.226.22]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 86EF91A094A; Mon, 5 Oct 2020 19:37:01 +0200 (CEST) Received: from fsr-ub1864-111.ea.freescale.net (fsr-ub1864-111.ea.freescale.net [10.171.82.141]) by inva024.eu-rdc02.nxp.com (Postfix) with ESMTP id 3FF202033F; Mon, 5 Oct 2020 19:37:01 +0200 (CEST) From: Diana Craciun To: Alex Williamson , kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, bharatb.linux@gmail.com, laurentiu.tudor@nxp.com, Diana Craciun Subject: [PATCH v6 10/10] vfio/fsl-mc: Add support for device reset Date: Mon, 5 Oct 2020 20:36:54 +0300 Message-Id: <20201005173654.31773-11-diana.craciun@oss.nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20201005173654.31773-1-diana.craciun@oss.nxp.com> References: <20201005173654.31773-1-diana.craciun@oss.nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Currently only resetting the DPRC container is supported which will reset all the objects inside it. Resetting individual objects is possible from the userspace by issueing commands towards MC firmware. Signed-off-by: Diana Craciun Reviewed-by: Eric Auger --- drivers/vfio/fsl-mc/vfio_fsl_mc.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index d95568cd8021..d009f873578c 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -217,6 +217,10 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, return -EINVAL; info.flags = VFIO_DEVICE_FLAGS_FSL_MC; + + if (is_fsl_mc_bus_dprc(mc_dev)) + info.flags |= VFIO_DEVICE_FLAGS_RESET; + info.num_regions = mc_dev->obj_desc.region_count; info.num_irqs = mc_dev->obj_desc.irq_count; @@ -299,7 +303,19 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd, } case VFIO_DEVICE_RESET: { - return -ENOTTY; + int ret; + struct fsl_mc_device *mc_dev = vdev->mc_dev; + + /* reset is supported only for the DPRC */ + if (!is_fsl_mc_bus_dprc(mc_dev)) + return -ENOTTY; + + ret = dprc_reset_container(mc_dev->mc_io, 0, + mc_dev->mc_handle, + mc_dev->obj_desc.id, + DPRC_RESET_OPTION_NON_RECURSIVE); + return ret; + } default: return -ENOTTY;