From patchwork Wed Apr 26 14:54:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 254C0C7EE22 for ; Wed, 26 Apr 2023 14:54:29 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3DD1B10E293; Wed, 26 Apr 2023 14:54:28 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 95C7510E39F; Wed, 26 Apr 2023 14:54:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520866; x=1714056866; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ei+ldlGBA0woJE13bWLVPf1qW3075XzsC5F91f4P4Ag=; b=FSY4ouGrW6qyIfEZgQaIB/Dyn+ASlIGiVbrQ2YIey+VJYjN0mGLRqjoZ 1qeOE11wVRxTcfWxm8rHsngjP4JXfEOk49sk/YCEOsk+vEDVG2WQx7sDi VJTRiffJAWiSaA9Jxbp1VmdnhX5zQ/gXLSuZ5BmhtvDMk1vURu4oJTems FRZcGWkVSCTCvIwVKTKXUf4tUiqPrP4VJ3nwmjEEEldrn+IyHq7Wz4ibG owjT9F2iePrinEGSrzUZ4keG5mn1ajZ+sG0ZkqAAi51hpPKwn//nBWRLo hb12ze0LJjquBqDUez+AjjuT171R9QKxJ7XEMJu/1xFBEHOFNQA66KT6W w==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410234" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410234" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758643988" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758643988" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:24 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:11 -0700 Message-Id: <20230426145419.450922-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 1/9] vfio: Determine noiommu in vfio_device registration X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This adds a noiommu flag in vfio_device, hence caller of the vfio_device_is_noiommu() just refers to the flag for noiommu check. Reviewed-by: Kevin Tian Tested-by: Nicolin Chen Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/iommufd.c | 4 ++-- drivers/vfio/vfio.h | 7 ++++--- drivers/vfio/vfio_main.c | 4 ++++ include/linux/vfio.h | 1 + 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 88b00c501015..895852ad37ed 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -18,7 +18,7 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx) lockdep_assert_held(&vdev->dev_set->lock); - if (vfio_device_is_noiommu(vdev)) { + if (vdev->noiommu) { if (!capable(CAP_SYS_RAWIO)) return -EPERM; @@ -59,7 +59,7 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) { lockdep_assert_held(&vdev->dev_set->lock); - if (vfio_device_is_noiommu(vdev)) + if (vdev->noiommu) return; if (vdev->ops->unbind_iommufd) diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 7b19c621e0e6..1ddf43863ad6 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -88,10 +88,11 @@ bool vfio_device_has_container(struct vfio_device *device); int __init vfio_group_init(void); void vfio_group_cleanup(void); -static inline bool vfio_device_is_noiommu(struct vfio_device *vdev) +static inline int vfio_device_set_noiommu(struct vfio_device *device) { - return IS_ENABLED(CONFIG_VFIO_NOIOMMU) && - vdev->group->type == VFIO_NO_IOMMU; + device->noiommu = IS_ENABLED(CONFIG_VFIO_NOIOMMU) && + device->group->type == VFIO_NO_IOMMU; + return 0; } #if IS_ENABLED(CONFIG_VFIO_CONTAINER) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 89497c933490..09be9df2ceca 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -276,6 +276,10 @@ static int __vfio_register_dev(struct vfio_device *device, if (ret) return ret; + ret = vfio_device_set_noiommu(device); + if (ret) + goto err_out; + ret = device_add(&device->device); if (ret) goto err_out; diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 2c137ea94a3e..4ee613924435 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -62,6 +62,7 @@ struct vfio_device { struct iommufd_device *iommufd_device; bool iommufd_attached; #endif + bool noiommu; }; /** From patchwork Wed Apr 26 14:54:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87ED3C7EE24 for ; Wed, 26 Apr 2023 14:54:40 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D173410E9D5; Wed, 26 Apr 2023 14:54:39 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 076A610E436; Wed, 26 Apr 2023 14:54:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520869; x=1714056869; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sdoYFPhd6sLPYPVG93DO+sP/RGUhPCk04kqwDwb+nRY=; b=gWUQy0Sx4SdCWnqExy1Wfu1gV+y6kb01sWoojm4vfMtZCBVC+9HZGufK 1fWRWVAF9MQwnVScQrhOnverjLAht/PyABr6HkJFd3aieK2EfGkXhWjfr 2KSTFgFyh+pwwOX+g0VV9rsnh6puGF7KHmB/G5RzjGGVd8ZCgk7TqJWQo 8j+ehpIkDaB8EB3IkFc/+ARfw3DGKwDlPRH3at1M0LI8DijoUrWHFu2FO gFYfPtmPLF2yOEXKhyuEFB9MPw+rvTjpp37ToREsNydMbPqi9pxeHCJ4g lr42tgyl+wUIiO/hrsGzpXg/hik6u7MzrgHCPLQ2NMP1fucqXSyHpktsV g==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410252" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410252" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644002" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644002" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:27 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:12 -0700 Message-Id: <20230426145419.450922-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 2/9] vfio-iommufd: Create iommufd_access for noiommu devices X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This binds noiommu device to iommufd and creates iommufd_access for this bond. This is useful for adding an iommufd-based device ownership check for VFIO_DEVICE_PCI_HOT_RESET since this model requires all the other affected devices bound to the same iommufd as the device to be reset. For noiommu devices, there is no backend iommu, so create iommufd_access instead of iommufd_device. Suggested-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/iommufd.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 895852ad37ed..ca29c4feded3 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -29,7 +29,8 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx) */ if (!iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id)) return -EPERM; - return 0; + + return vfio_iommufd_emulated_bind(vdev, ictx, &device_id); } ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id); @@ -59,8 +60,10 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) { lockdep_assert_held(&vdev->dev_set->lock); - if (vdev->noiommu) + if (vdev->noiommu) { + vfio_iommufd_emulated_unbind(vdev); return; + } if (vdev->ops->unbind_iommufd) vdev->ops->unbind_iommufd(vdev); @@ -110,10 +113,14 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id) EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas); /* - * The emulated standard ops mean that vfio_device is going to use the - * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this - * ops set should call vfio_register_emulated_iommu_dev(). Drivers that do - * not call vfio_pin_pages()/vfio_dma_rw() have no need to provide dma_unmap. + * The emulated standard ops can be used by below usages: + * 1) The vfio_device that is going to use the "mdev path" and will call + * vfio_pin_pages()/vfio_dma_rw(). Such drivers using should call + * vfio_register_emulated_iommu_dev(). Drivers that do not call + * vfio_pin_pages()/vfio_dma_rw() have no need to provide dma_unmap. + * 2) The noiommu device which doesn't have backend iommu but creating + * an iommufd_access allows generating a dev_id for it. noiommu device + * is not allowed to do map/unmap so this becomes a nop. */ static void vfio_emulated_unmap(void *data, unsigned long iova, @@ -121,7 +128,8 @@ static void vfio_emulated_unmap(void *data, unsigned long iova, { struct vfio_device *vdev = data; - if (vdev->ops->dma_unmap) + /* noiommu devices cannot do map/unmap */ + if (vdev->noiommu && vdev->ops->dma_unmap) vdev->ops->dma_unmap(vdev, iova, length); } From patchwork Wed Apr 26 14:54:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 50FB4C7EE22 for ; Wed, 26 Apr 2023 14:54:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B718C10E39F; Wed, 26 Apr 2023 14:54:35 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id B30FA10E436; Wed, 26 Apr 2023 14:54:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520869; x=1714056869; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aC0EHWzpMalEv7sDoXtuog1stFMUCxCIBPha87twB+0=; b=eV8q25Pn2mmv2K6uXVVa2iiXp4lIuxlRJS/iNAB360X4hPB7BJI0Bpsy F8kqXEUFJJ6Ut57J8sX5Jy9cFKyQbIDvXsAhNLsVgE2g2LEgGPClC6cPx 4qeR2cROlViu5RvAK0FZW6M4loLyypREKdfkKB1IAg9Tc/FhHDCaxv5zp 9xdaN/xtscWT/12VNrw55olj0mWiA34tsr/fCkpD9hGpnbKqFddfDf72R EDDTXD0KVQvE+TCifc7lUbV3UqHeGEqVdq6PGHvccIeC+OSWP2ksCcGHP SIMbEJEXfMIbuov3fgIFPPcjXS61CnYrQNyJGQ54Ygd0hh081+a9xt2zI Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410268" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410268" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644007" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644007" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:28 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:13 -0700 Message-Id: <20230426145419.450922-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 3/9] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Reviewed-by: Eric Auger Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..f824de4dbf27 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group is held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Wed Apr 26 14:54:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224674 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79DA6C7EE22 for ; Wed, 26 Apr 2023 14:54:45 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 153EC10E694; Wed, 26 Apr 2023 14:54:45 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id F04FE10E39F; Wed, 26 Apr 2023 14:54:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520874; x=1714056874; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F/+2vNJVk+zK6U7ggeqab7eTb50HgDYfR/ztN//ugrU=; b=TsJiadvsZDLQrGIQd9bzJ+yC+c2DovOxYUaUU2T+GPVKqbQI8m7DfW97 4kQaNN7o9XkmfoCmk7DCGeTuyn0TviIpGeTlOedcALZ+dCA/fBppPCh7s QyZIHtrbEJEg2YMhzJKbCtTYxozr93GioE6cR14FPvLlwBorrEeiJ8JL1 HU3srx0RzGtxhX1rlgn2R1kcJYfqGw8gjW2/3lDBXSQnia4vUWJ40DeGs zgciKp6BZ6BrMrApxjvQmrJ8bzaj2KETDea2sHdXKshoTpsSXy2lWUgoG l13PWjVNH8AuyjDY4hf4BebbKINiv0cYskDCyaQ/x/7IGA6shP/B/WXOs g==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410306" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410306" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644025" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644025" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:32 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:14 -0700 Message-Id: <20230426145419.450922-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 4/9] vfio/pci: Move the existing hot reset logic to be a helper X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups(). No functional change is intended. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Eric Auger Tested-by: Yanting Jiang Signed-off-by: Yi Liu Reviewed-by: Kevin Tian --- drivers/vfio/pci/vfio_pci_core.c | 55 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index f824de4dbf27..39e7823088e7 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1255,29 +1255,16 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + int array_count, bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1276,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (!array_count || array_count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(array_count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1288,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + array_count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1298,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group is held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < array_count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,7 +1322,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = array_count; info.files = files; ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); @@ -1348,6 +1335,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { From patchwork Wed Apr 26 14:54:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBB63C77B78 for ; Wed, 26 Apr 2023 14:54:38 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6533110E90A; Wed, 26 Apr 2023 14:54:38 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id F2B0810E39F; Wed, 26 Apr 2023 14:54:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520875; x=1714056875; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=io8hL9bTUpAjOWSG+LYha6CPKcMkhPDUJOxDgMz4F3c=; b=di0q9kjddOmMibLKyv3hwMmpwoEqs2DfvTGMsVHkLGHkcpRXOOVphFi1 LnclwTjfu1iXZtDVGrHN4vvQJbcrtQfXCSLyPxkO1tMlN1/UarYgmgBb7 S68Ei4h/gJ567LigzfYgyF3mV3CTAV7c2aWLvhTWxgWl+53ztTljiFPxF ND5hKFL85CQKEq9TfTVViraiLMkWEHkY9OZ4jNJ2g7BJvHXEFzVtcm1bU MEzWSoXdluBVQdYnq6IZWsxNKIU7mLzHtzuL6nhi7vnsTAADgWrdMQQXK QfqPR0cjh6AGhtH1J7vmMok1ex6OqRDMMY+WiV53wmC/YwViy1K6240CC w==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410320" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410320" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644033" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644033" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:34 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:15 -0700 Message-Id: <20230426145419.450922-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 5/9] vfio: Mark cdev usage in vfio_device X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Use it to differentiate whether to report group_id or dev_id in revised VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl. Though it is not set at this moment introducing it now allows us to get hot reset ready for cdev. Signed-off-by: Yi Liu Reviewed-by: Kevin Tian --- include/linux/vfio.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 4ee613924435..298f4ef16be7 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -63,6 +63,7 @@ struct vfio_device { bool iommufd_attached; #endif bool noiommu; + bool cdev_opened; }; /** @@ -140,6 +141,12 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) #endif +static inline bool vfio_device_cdev_opened(struct vfio_device *device) +{ + lockdep_assert_held(&device->dev_set->lock); + return device->cdev_opened; +} + /** * struct vfio_migration_ops - VFIO bus device driver migration callbacks * From patchwork Wed Apr 26 14:54:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C4CE2C77B7C for ; Wed, 26 Apr 2023 14:54:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 355FF10E530; Wed, 26 Apr 2023 14:54:44 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 714C810E530; Wed, 26 Apr 2023 14:54:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520876; x=1714056876; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=we3Zabe+6AgPN8c/FPEQV6hmIff3h/EBF2BbVZP1HZQ=; b=O7vDQNbhUKyujEmrsuyKmfTlfAmgelud2flgcv/zAoXm1q/icHDXcKss 8EsMsCUy866Xxqz0J9rcmVo9Z+bfSrIz1+rIizPsbfXkqPnSwlSpywS16 fB302FqKQdXByDt/yMLVWaTztG2yW6CHHUZIFWPKC6Rskzp9FfABLnK1R 8gVkGYT2/XEB2U3coSkRNEnxKwM1dxuXUNRZZ8KulabOygrGzxxRTQPGB 0TOos/YR5NM0aP9V/EfhzdTp1qfywoHXD3ja8PXJ9ddcOeHXUPHNT3Qao io5LsXNGVIgpMzFKTH/VYmLZOQjbGwBjyqn0CUlRRcKEDbBQkYNouiD0X g==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410337" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410337" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644040" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644040" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:35 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:16 -0700 Message-Id: <20230426145419.450922-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 6/9] iommufd: Reserved -1 in the iommufd xarray X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" VFIO needs two reserved values. 0 is already reserved by initializing xarray with XA_FLAGS_ALLOC1. This reserves -1 by limiting the xa alloc range. Signed-off-by: Yi Liu --- drivers/iommu/iommufd/main.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c index 3fbe636c3d8a..51b27c96c52f 100644 --- a/drivers/iommu/iommufd/main.c +++ b/drivers/iommu/iommufd/main.c @@ -28,6 +28,9 @@ struct iommufd_object_ops { static const struct iommufd_object_ops iommufd_object_ops[]; static struct miscdevice vfio_misc_dev; +/* -1 is reserved */ +#define iommufd_xa_limit_32b XA_LIMIT(0, (-2U)) + struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, size_t size, enum iommufd_object_type type) @@ -50,7 +53,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, * before calling iommufd_object_finalize(). */ rc = xa_alloc(&ictx->objects, &obj->id, XA_ZERO_ENTRY, - xa_limit_32b, GFP_KERNEL_ACCOUNT); + iommufd_xa_limit_32b, GFP_KERNEL_ACCOUNT); if (rc) goto out_free; return obj; From patchwork Wed Apr 26 14:54:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C736C77B7C for ; Wed, 26 Apr 2023 14:54:50 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DF2B010E9DB; Wed, 26 Apr 2023 14:54:49 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7814B10E7A3; Wed, 26 Apr 2023 14:54:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520877; x=1714056877; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=W/ChmFWvOJoBSp4nMBstR02MUyL97oP5XH+845iXrIM=; b=girdFCGsbsJy4MBF94sOOmvNX6fuY59LNVj5fONkA0M8aE/HK2rP80i7 mYkHfNWS+BxWa1El4JNvGV1mlit5Rwjhyt7RX+wlCT6ih/LwZLPrc539o ApIA8l3IxmcZtIAKa3jyneNwnJ/fnB6MMqPJP81S0bZxgNUKzd82nfWcG hUEjedDIm7PDNEJf6jBZ0IHgnpeQdrAWIucQ/Rz26BMDLdmc6JHVv/bA4 Owftf4eivsThe63l8tBrInHiU9cqeBcQcPvSimiGc0fk8DuKNQOGtGj39 arpR6kFeidwQFSmtPInh4K8F+j8deRye6CpHKzXwMRvt4/7Jd3EAsX00n w==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410347" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410347" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644051" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644051" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:36 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:17 -0700 Message-Id: <20230426145419.450922-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 7/9] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device. Signed-off-by: Yi Liu --- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 24 ++++++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 68 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 4b565a5b51da..40b52f1a071d 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + +u32 iommufd_device_to_id(struct iommufd_device *idev) +{ + return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) @@ -480,6 +492,18 @@ void iommufd_access_destroy(struct iommufd_access *access) } EXPORT_SYMBOL_NS_GPL(iommufd_access_destroy, IOMMUFD); +struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access) +{ + return access->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_access_to_ictx, IOMMUFD); + +u32 iommufd_access_to_id(struct iommufd_access *access) +{ + return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_access_to_id, IOMMUFD); + int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id) { struct iommufd_ioas *new_ioas; diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index ca29c4feded3..54ac7be49b80 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -69,6 +69,30 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (vdev->iommufd_device) + return iommufd_device_to_ictx(vdev->iommufd_device); + if (vdev->iommufd_access) + return iommufd_access_to_ictx(vdev->iommufd_access); + return NULL; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + +/* + * Return devid for devices that have been bound with iommufd, + * returns 0 if not bound yet. + */ +u32 vfio_iommufd_physical_devid(struct vfio_device *vdev) +{ + if (WARN_ON(!vdev->iommufd_device && !vdev->iommufd_access)) + return 0; + if (vdev->iommufd_device) + return iommufd_device_to_id(vdev->iommufd_device); + else + return iommufd_access_to_id(vdev->iommufd_access); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid); /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..68cd65274e28 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); +u32 iommufd_device_to_id(struct iommufd_device *idev); + struct iommufd_access_ops { u8 needs_pin_pages : 1; void (*unmap)(void *data, unsigned long iova, unsigned long length); @@ -45,6 +48,9 @@ iommufd_access_create(struct iommufd_ctx *ictx, void iommufd_access_destroy(struct iommufd_access *access); int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id); +struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access); +u32 iommufd_access_to_id(struct iommufd_access *access); + void iommufd_ctx_get(struct iommufd_ctx *ictx); #if IS_ENABLED(CONFIG_IOMMUFD) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 298f4ef16be7..cead7aebd527 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -116,6 +116,8 @@ struct vfio_device_ops { }; #if IS_ENABLED(CONFIG_IOMMUFD) +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); +u32 vfio_iommufd_physical_devid(struct vfio_device *vdev); int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); @@ -125,6 +127,18 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev, void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); #else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + return NULL; +} + +static inline u32 +vfio_iommufd_physical_devid(struct vfio_device *vdev) +{ + return 0; +} + #define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Wed Apr 26 14:54:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18F43C7618E for ; Wed, 26 Apr 2023 14:54:55 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 361A310E9E8; Wed, 26 Apr 2023 14:54:54 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A19B10E9DA; Wed, 26 Apr 2023 14:54:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520880; x=1714056880; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J7uU8ORTy0thRx4RxPcQ82ksVnUJ4EqDibdEuTyIzGs=; b=FoP0ZGjlQ+Y+j+dE3BsBoPEoj0SMwPL5v80XQBKhQUiAp3PwQ2yvMxHB f1010PAxduknNDHl35yf/2zxAKl3EWuEeB//xZKanoHPTJr8CrkiN74J5 hEB+GALVSg4l1tUAeqmjK/iQS0AhlGMYqax0ICsMJU3AwVn0dBcz7Ii5b AhWMSEGBfqq6u6BncH0L1sn4/fzzKvidokTXKpNs7xoUQKQLLMo6t2O4U n9a3vDwtRLuZL+ETZ2inxnTSUpySkiU+9aIabXVsXIA6o7zUUCpYNzfzw uVxuqqaBzLlIvqJvC9VVnDjLIgks5Z9e9MsUuwZ0d2LtKPt0vkRQltC/Y A==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410364" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410364" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644064" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644064" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:39 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:18 -0700 Message-Id: <20230426145419.450922-9-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 8/9] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This makes VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl to use the bound iommufd of the cdev device to check the ownership of the other affected devices and set a flag to tell user if the cdev device is resettable with a zero-length fd array. For each of the affected devices, if it is bound to the iommufd of the cdev device, _INFO reports a valid dev_id > 0; if it is not opened by the calling user, but it is in the iommu_group of a device that is bound to the iommufd of the cdev device, reports dev_id == 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the _INFO ioctl shall report dev_id==-1 for such affected devices. dev_id >=0 doesn't block hot-reset, while dev_id == -1 will block hot-reset. This adds flag VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID to tell the user dev_id is returned and adds flag VFIO_PCI_HOT_RESET_FLAG_RESETTABLE to tell user if the cdev device is resettable or not. Suggested-by: Jason Gunthorpe Suggested-by: Alex Williamson Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 101 ++++++++++++++++++++++++++++--- include/uapi/linux/vfio.h | 39 +++++++++++- 2 files changed, 132 insertions(+), 8 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 39e7823088e7..43858d471447 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -766,6 +766,51 @@ static int vfio_pci_get_irq_count(struct vfio_pci_core_device *vdev, int irq_typ return 0; } +static struct vfio_device * +vfio_pci_find_device_in_devset(struct vfio_device_set *dev_set, + struct pci_dev *pdev) +{ + struct vfio_device *cur; + + lockdep_assert_held(&dev_set->lock); + + list_for_each_entry(cur, &dev_set->device_list, dev_set_list) + if (cur->dev == &pdev->dev) + return cur; + return NULL; +} + +/* + * Check if a given iommu_group has been bound to an iommufd within a + * devset. Returns true if there is device in the devset which is in + * the input iommu_group and meanwhile bound to the input iommufd. + * Otherwise, returns false. + */ +static bool +vfio_devset_iommufd_has_group(struct vfio_device_set *dev_set, + struct iommufd_ctx *iommufd, + struct iommu_group *iommu_group) +{ + struct vfio_device *cur; + struct iommu_group *grp; + bool found = false; + + lockdep_assert_held(&dev_set->lock); + + list_for_each_entry(cur, &dev_set->device_list, dev_set_list) { + grp = iommu_group_get(cur->dev); + if (!grp) + continue; + iommu_group_put(grp); + if (iommu_group == grp && + iommufd == vfio_iommufd_physical_ictx(cur)) { + found = true; + break; + } + } + return found; +} + static int vfio_pci_count_devs(struct pci_dev *pdev, void *data) { (*(int *)data)++; @@ -776,13 +821,20 @@ struct vfio_pci_fill_info { int max; int cur; struct vfio_pci_dependent_device *devices; + struct vfio_device *vdev; + bool devid; + bool resettable; }; static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) { struct vfio_pci_fill_info *fill = data; + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(fill->vdev); + struct vfio_device_set *dev_set = fill->vdev->dev_set; struct iommu_group *iommu_group; + lockdep_assert_held(&dev_set->lock); + if (fill->cur == fill->max) return -EAGAIN; /* Something changed, try again */ @@ -790,7 +842,34 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) if (!iommu_group) return -EPERM; /* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + if (fill->devid) { + struct vfio_device *vdev; + + /* + * Report devid for the affected devices: + * - valid devid > 0 for the devices that are bound with + * the iommufd of the calling device. + * - devid == 0 for the devices that have not been opened + * but have same group with one of the devices bound to + * the iommufd of the calling device. + * - devid == -1 for others, and clear resettable flag. + */ + vdev = vfio_pci_find_device_in_devset(dev_set, pdev); + if (vdev && iommufd == vfio_iommufd_physical_ictx(vdev)) { + fill->devices[fill->cur].dev_id = + vfio_iommufd_physical_devid(vdev); + if (unlikely(!fill->devices[fill->cur].dev_id)) + return -EINVAL; + } else if (vfio_devset_iommufd_has_group(dev_set, iommufd, + iommu_group)) { + fill->devices[fill->cur].dev_id = VFIO_PCI_DEVID_NONBLOCKING; + } else { + fill->devices[fill->cur].dev_id = VFIO_PCI_DEVID_BLOCKING; + fill->resettable = false; + } + } else { + fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + } fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn; @@ -1229,17 +1308,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return -ENOMEM; fill.devices = devices; + fill.vdev = &vdev->vdev; + mutex_lock(&vdev->vdev.dev_set->lock); + fill.devid = fill.resettable = vfio_device_cdev_opened(&vdev->vdev); ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_fill_devs, &fill, slot); + mutex_unlock(&vdev->vdev.dev_set->lock); /* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ - if (!ret) + if (!ret) { hdr.count = fill.cur; + if (fill.devid) { + hdr.flags = VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID; + if (fill.resettable) + hdr.flags |= VFIO_PCI_HOT_RESET_FLAG_RESETTABLE; + } + } reset_info_exit: if (copy_to_user(arg, &hdr, minsz)) @@ -2335,12 +2424,10 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, static int vfio_pci_is_device_in_set(struct pci_dev *pdev, void *data) { struct vfio_device_set *dev_set = data; - struct vfio_device *cur; - list_for_each_entry(cur, &dev_set->device_list, dev_set_list) - if (cur->dev == &pdev->dev) - return 0; - return -EBUSY; + lockdep_assert_held(&dev_set->lock); + + return vfio_pci_find_device_in_devset(dev_set, pdev) ? 0 : -EBUSY; } /* diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..4b4e2c28984b 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -650,11 +650,46 @@ enum { * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * + * This command is used to query the affected devices in the hot reset for + * a given device. + * + * This command always reports the segment, bus, and devfn information for + * each affected device, and selectively reports the group_id or dev_id per + * the way how the calling device is opened. + * + * - If the calling device is opened via the traditional group/container + * API, group_id is reported. User should check if it has owned all + * the affected devices and provides a set of group fds to prove the + * ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. + * + * - If the calling device is opened as a cdev, dev_id is reported. + * Flag VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID would be set. Flag + * VFIO_PCI_HOT_RESET_FLAG_RESETTABLE would be set per the ownership + * of the other affected devices. If it is set, the user could invoke + * VFIO_DEVICE_PCI_HOT_RESET with a zero-length fd array. Kernel + * set this flag when all the affected devices are owned by the user. + * This flag is available only VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID + * is set, otherwise ignored. For a given affected device, it is owned + * if it suits one of the below cases: + * 1) bound to the same iommufd_ctx with the calling device + * 2) has not been bound to iommufd_ctx, but it is within the + * iommu_group of an owned device. + * For 1), the dev_id > 0, for 2) dev_id == 0. Otherwise, dev_id == -1. + * + * If the affected devices of a calling device span into multiple iommufds + * or opened by different APIs (group/container or cdev), hot-reset on + * this device would be rejected. + * * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { - __u32 group_id; + union { + __u32 group_id; + __u32 dev_id; +#define VFIO_PCI_DEVID_NONBLOCKING 0 +#define VFIO_PCI_DEVID_BLOCKING -1 + }; __u16 segment; __u8 bus; __u8 devfn; /* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +698,8 @@ struct vfio_pci_dependent_device { struct vfio_pci_hot_reset_info { __u32 argsz; __u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_RESETTABLE (1 << 1) __u32 count; struct vfio_pci_dependent_device devices[]; }; From patchwork Wed Apr 26 14:54:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13224676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2CA2EC77B78 for ; Wed, 26 Apr 2023 14:54:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C474C10E9E4; Wed, 26 Apr 2023 14:54:53 +0000 (UTC) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by gabe.freedesktop.org (Postfix) with ESMTPS id E671510E530; Wed, 26 Apr 2023 14:54:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682520882; x=1714056882; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2K7nvdIG5YIxPwK2ZtjJpN2nWmwhSE9eiTVKfvbiR/Y=; b=bIf/zHTuzMc/mSknU/ijMG/HNNbyCHDs0XY7tLUYobwipBHHiUzkH47T YffiuddVCbabu5gAiWS7ZHTLkMTRxAyrz8DBKYYjlpl6Hf295g7tnNxN2 AF184O9PlqF3whk4U8+gOWWMfoLekcvPpL9CpkrdhFaA/ZguHJ/mygahk gU+7fMwWwYUK3h44Sx7IwJeAlfmr6gl7t5o5M35pm1Jl47T7UZtNBR04r VyG7PHFlSkS9krWBgyvNbboddda2jszW8LyBY+0osUc4LstNzI7/+nAib ckF4iyiNlalXXEkTYhisD3BjNtYspYSxVRkMOeIJJ5TzXkWBIy18kbl6D w==; X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="433410377" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="433410377" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2023 07:54:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10692"; a="758644069" X-IronPort-AV: E=Sophos;i="5.99,228,1677571200"; d="scan'208";a="758644069" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by fmsmga008.fm.intel.com with ESMTP; 26 Apr 2023 07:54:40 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Date: Wed, 26 Apr 2023 07:54:19 -0700 Message-Id: <20230426145419.450922-10-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230426145419.450922-1-yi.l.liu@intel.com> References: <20230426145419.450922-1-yi.l.liu@intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH v4 9/9] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mjrosato@linux.ibm.com, jasowang@redhat.com, xudong.hao@intel.com, zhenzhong.duan@intel.com, peterx@redhat.com, terrence.xu@intel.com, chao.p.peng@linux.intel.com, linux-s390@vger.kernel.org, yi.l.liu@intel.com, kvm@vger.kernel.org, lulu@redhat.com, yanting.jiang@intel.com, joro@8bytes.org, nicolinc@nvidia.com, yan.y.zhao@intel.com, intel-gfx@lists.freedesktop.org, eric.auger@redhat.com, intel-gvt-dev@lists.freedesktop.org, yi.y.sun@linux.intel.com, cohuck@redhat.com, shameerali.kolothum.thodi@huawei.com, suravee.suthikulpanit@amd.com, robin.murphy@arm.com Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_RESETTABLE in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 66 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 22 +++++++++++ 2 files changed, 79 insertions(+), 9 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 43858d471447..f70e3b948b16 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1364,8 +1365,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (ret) return ret; - /* Somewhere between 1 and count is OK */ - if (!array_count || array_count > count) + if (array_count > count) return -EINVAL; group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL); @@ -1414,7 +1414,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = array_count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1429,6 +1429,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, { unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); struct vfio_pci_hot_reset hdr; + struct iommufd_ctx *iommufd; bool slot = false; if (copy_from_user(&hdr, arg, minsz)) @@ -1443,7 +1444,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, else if (pci_probe_reset_bus(vdev->pdev->bus)) return -ENODEV; - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, @@ -2415,6 +2421,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) return true; @@ -2488,13 +2497,38 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + struct iommu_group *iommu_group; + + if (!iommufd_ctx) + return false; + + if (iommufd == iommufd_ctx) + return true; + + iommu_group = iommu_group_get(vdev->vdev.dev); + if (!iommu_group) + return false; + + /* + * Try to check if any device within iommu_group is bound with + * the input iommufd_ctx. + */ + return vfio_devset_iommufd_has_group(vdev->vdev.dev_set, + iommufd_ctx, iommu_group); +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2525,10 +2559,24 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * If user provides a set of groups, all the opened devices + * in the dev_set should be contained by the set of groups + * provided by the user. + * + * If user provides a zero-length group fd array, then all + * the affected devices must be bound to same iommufd_ctx as + * the input iommufd_ctx. If there is device that has not + * been bound to iommufd_ctx yet, shall check if there is any + * device within its iommu_group that has been bound to the + * input iommufd_ctx. + * + * Otherwise, reset is not allowed. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (!vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 4b4e2c28984b..1241d02d8701 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -710,6 +710,28 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it operates. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices are owned by itself. + * The ownership proof needs to refer the output of + * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. Ownership can be proved as: + * + * 1) An array of group fds - This is used for the devices opened via + * the group/container interface. + * 2) A zero-length array - This is used for the devices opened via + * the cdev interface. User should check the + * flag VFIO_PCI_HOT_RESET_FLAG_IOMMUFD_DEV_ID + * and flag VFIO_PCI_HOT_RESET_FLAG_RESETTABLE + * before using this method. + * + * In case a non void group fd array is passed, the devices affected by + * the reset must belong to those opened VFIO groups. In case a zero + * length array is passed, the other devices affected by the reset, if + * any, must be either bound to the same iommufd as this VFIO device or + * in the same iommu_group with a device that does. Either of the two + * methods is applied to check the feasibility of the hot reset. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset {