From patchwork Sat May 13 13:21:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECF1FC7EE2D for ; Sat, 13 May 2023 13:21:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237665AbjEMNVn (ORCPT ); Sat, 13 May 2023 09:21:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233967AbjEMNVl (ORCPT ); Sat, 13 May 2023 09:21:41 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 808393C1D; Sat, 13 May 2023 06:21:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984100; x=1715520100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7Jud81j68GXwp1AZyUJqqegxBroEGFVO9r8F7tMxbEk=; b=VmnZAbkWd6jxEEfnxQQHRv6VQAIG99jzkrr54FzcoftnaE819hs6Jj6N SUKS7nQiobqHwrKrwKWCzuW+EwmsGs5mBrgPlWB8HoqpMkRW3+sDA7muz EwjIyQ7KaLaFyzildCq6oHHWgDGGbthQHQfuahzpTVXVPATb5iHWG3Cn6 AdV85G2dsJigy69aAmCdtj+WQZLs+WphyA5y16i4mnhbBEFOWf5QzBSE+ VpzEzzfzaLnGv9mE5qcOR8nPgW+9fBP7omBDTbnAPWhqKzMm7Fa+Df4Dl ICp9XEqbzlPCNfHBVv0kLdYhVzwja3lvYzi9wTJ59k/Xwz7yc/Ks3zMIw A==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416598947" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416598947" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126430" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126430" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:39 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 01/10] vfio-iommufd: Create iommufd_access for noiommu devices Date: Sat, 13 May 2023 06:21:27 -0700 Message-Id: <20230513132136.15021-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This binds noiommu device to iommufd and creates iommufd_access for this bond. This is useful for adding an iommufd-based device ownership check for VFIO_DEVICE_PCI_HOT_RESET since this model requires all the other affected devices bound to the same iommufd as the device to be reset. For noiommu devices, there is no backend iommu, so create iommufd_access instead of iommufd_device. Suggested-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/iommufd.c | 43 ++++++++++++++++++++++++++++++++++++++++-- include/linux/vfio.h | 1 + 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 88b00c501015..c1379e826052 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -10,6 +10,42 @@ MODULE_IMPORT_NS(IOMMUFD); MODULE_IMPORT_NS(IOMMUFD_VFIO); +static void vfio_noiommu_access_unmap(void *data, unsigned long iova, + unsigned long length) +{ +} + +static const struct iommufd_access_ops vfio_user_noiommu_ops = { + .needs_pin_pages = 1, + .unmap = vfio_noiommu_access_unmap, +}; + +static int vfio_iommufd_noiommu_bind(struct vfio_device *vdev, + struct iommufd_ctx *ictx, + u32 *out_device_id) +{ + struct iommufd_access *user; + + lockdep_assert_held(&vdev->dev_set->lock); + + user = iommufd_access_create(ictx, &vfio_user_noiommu_ops, + vdev, out_device_id); + if (IS_ERR(user)) + return PTR_ERR(user); + vdev->noiommu_access = user; + return 0; +} + +static void vfio_iommufd_noiommu_unbind(struct vfio_device *vdev) +{ + lockdep_assert_held(&vdev->dev_set->lock); + + if (vdev->noiommu_access) { + iommufd_access_destroy(vdev->noiommu_access); + vdev->noiommu_access = NULL; + } +} + int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx) { u32 ioas_id; @@ -29,7 +65,8 @@ int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx) */ if (!iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id)) return -EPERM; - return 0; + + return vfio_iommufd_noiommu_bind(vdev, ictx, &device_id); } ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id); @@ -59,8 +96,10 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) { lockdep_assert_held(&vdev->dev_set->lock); - if (vfio_device_is_noiommu(vdev)) + if (vfio_device_is_noiommu(vdev)) { + vfio_iommufd_noiommu_unbind(vdev); return; + } if (vdev->ops->unbind_iommufd) vdev->ops->unbind_iommufd(vdev); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 2c137ea94a3e..16fd04490550 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -57,6 +57,7 @@ struct vfio_device { struct list_head group_next; struct list_head iommu_entry; struct iommufd_access *iommufd_access; + struct iommufd_access *noiommu_access; void (*put_kvm)(struct kvm *kvm); #if IS_ENABLED(CONFIG_IOMMUFD) struct iommufd_device *iommufd_device; From patchwork Sat May 13 13:21:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59B6FC7EE2E for ; Sat, 13 May 2023 13:21:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237852AbjEMNVp (ORCPT ); Sat, 13 May 2023 09:21:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232564AbjEMNVm (ORCPT ); Sat, 13 May 2023 09:21:42 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12839E77; Sat, 13 May 2023 06:21:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984102; x=1715520102; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aC0EHWzpMalEv7sDoXtuog1stFMUCxCIBPha87twB+0=; b=RtekA7fhnvFxHQ+RDueFkoH50H5ZquXJDumQL1LMgWzc9hGuwrHdu856 w9R65Qj2ecUYGkKGIhC3ecD0mm/PFO/JnYD0GvllPc97E1HY2er+CRylI 20PGmB2QGLe1yCAemBpKRqtXs7Pvgsg4bcNuxMWwxkqrHun+FONHN/tNk TaOQfxtfgJMCD0KVo58hxZrmNwm3rmIRDeETnidJCA8j/wMngV/bIqgzc 5Gw9N3SL+AQa5Su98CcJFXYTKAZEN/7Sv2xCxUaHMyH4Buz1BiZFZMfpt QfXmPHeA4PR/c7vuZBMFF7QAh7WL+OCjrNaGJSskxcLpTISpqu6OG8R8W g==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416598960" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416598960" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126438" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126438" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:40 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 02/10] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() Date: Sat, 13 May 2023 06:21:28 -0700 Message-Id: <20230513132136.15021-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Reviewed-by: Eric Auger Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..f824de4dbf27 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group is held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Sat May 13 13:21:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B10FC7EE24 for ; Sat, 13 May 2023 13:21:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238077AbjEMNVr (ORCPT ); Sat, 13 May 2023 09:21:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232079AbjEMNVp (ORCPT ); Sat, 13 May 2023 09:21:45 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6B553C20; Sat, 13 May 2023 06:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984103; x=1715520103; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fW+4OuOU0THyl3K861KGJosgNhJujFuxa+w2SR7QIjQ=; b=bQBzbXH29hyEIgcu7WIxx3TLP7DDeU4NbNySqmunf0CMSfTEYCn3bOub dCW4rtDaUEJDHXFdaIARlk+H5L/yn9lvYUH8fYU6mGRtlOTFr6LPYG5lp vPOAW/H4dYmJ/tWtfl84FPVywaLaFav/JJ7BlGyzgkxF+Jfp34Z5uzlK/ m3aH1hZ7o+SSrlcVAkDmIgjvQE270HgItKP7bFrEEFlvrOH8DI/RhyQ8B T9EM06TRoC6Hy07IS/18uTf5plVSAZUoERbroniIUTnklA7JsFFcoCQ8S Ogl/r205bJKcaHpFmJbVkj5UdxHpp4R65oD41+XQahwxFMSH7O0Ny3bYq g==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416598970" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416598970" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126442" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126442" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:42 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 03/10] vfio/pci: Move the existing hot reset logic to be a helper Date: Sat, 13 May 2023 06:21:29 -0700 Message-Id: <20230513132136.15021-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups(). No functional change is intended. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Eric Auger Reviewed-by: Kevin Tian Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 55 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index f824de4dbf27..39e7823088e7 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1255,29 +1255,16 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + int array_count, bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1276,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (!array_count || array_count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(array_count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1288,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + array_count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1298,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group is held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < array_count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,7 +1322,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = array_count; info.files = files; ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); @@ -1348,6 +1335,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { From patchwork Sat May 13 13:21:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60D20C7EE25 for ; Sat, 13 May 2023 13:21:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238193AbjEMNVs (ORCPT ); Sat, 13 May 2023 09:21:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237878AbjEMNVq (ORCPT ); Sat, 13 May 2023 09:21:46 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D6A540EA; Sat, 13 May 2023 06:21:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984105; x=1715520105; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qN4HM5ImPoRe5kczhtMbmX4n3XHxYESenqa+cBcxH44=; b=bA+hmX7UPhduKuPVqTf+52+yBen/jFN/5QdSjwenkIl1HYfgXen1f3ev 8fRLD8s3b9miJjzDZdsibO147ecRP+CJnenXz+2aAsOfTjuK7a0PN838S n3y5xHot6qc4w9Z9MiFTTuwyBeI1y0Eks4erhNoczRILMaACph5AYRCpM hU2BVROJQVRUX2fKFWlOMB5ZzWCfUtTNkZcDMx/n7hoZL6gNEzoOOETn+ TeVbKfSCviBDa5YhJxVO0aKUyDEXBzg2JNqeiWJGVWLvkQA9FXvC5BvAm /J0j9ErNzDpsy73Wkf/0FxhqBu5POs7TIbgdqmBOvKEnrEeIZpootLQhA A==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416598981" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416598981" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:44 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126451" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126451" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:43 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 04/10] vfio: Mark cdev usage in vfio_device Date: Sat, 13 May 2023 06:21:30 -0700 Message-Id: <20230513132136.15021-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use it to differentiate whether to report group_id or devid in the revised VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl. At this moment, no cdev path yet, so the vfio_device_cdev_opened() helper always returns false. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu --- include/linux/vfio.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 16fd04490550..a61130bc06a2 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -140,6 +140,11 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) #endif +static inline bool vfio_device_cdev_opened(struct vfio_device *device) +{ + return false; +} + /** * struct vfio_migration_ops - VFIO bus device driver migration callbacks * From patchwork Sat May 13 13:21:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF0C5C77B7D for ; Sat, 13 May 2023 13:21:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238458AbjEMNVu (ORCPT ); Sat, 13 May 2023 09:21:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238125AbjEMNVs (ORCPT ); Sat, 13 May 2023 09:21:48 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6C0A19BC; Sat, 13 May 2023 06:21:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984106; x=1715520106; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ughe9Kcl9gMYisiLlx4VKGCel5QfZCognMPvf6ESs4U=; b=QiG1ojnlelEoi1DT9SgZKler+JrT7qDvSeugHKjiPmv+pfeOb0bCZi9E kSGz8M58fBd5+9hZjqGAtgzn5pL8TlCGTduzae3B7gKES0xrnwUuSyGiL jbck/2U1ZESAHcWqXtMbZIy0X/PWngIxpGZfUAxVj9PjVxhLemkqif/Pa NEciBKk1GXrz7+5AzqQaCHdVryLB8VmkM0IWGm9u5fRw2lTvgVS2kpMRC jGN5Ojz3erRi7Okam/qS9dTPtrmyyIQYLNqQx3Ik4vuqbu0sqIIHRKPSD eyHizVfxL3jRDiFuOmZNP1qzExHV5VJ7OE0Pesj+MzQPoTOh9Q2Ehech+ w==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416598995" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416598995" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126455" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126455" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:45 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 05/10] iommufd: Reserve all negative IDs in the iommufd xarray Date: Sat, 13 May 2023 06:21:31 -0700 Message-Id: <20230513132136.15021-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hence IOMMUFD users can encode the negative IDs for specific purposes. e.g. VFIO needs two reserved values to tell userspace the ID returned is not valid but has other meaning. Signed-off-by: Yi Liu --- drivers/iommu/iommufd/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/iommu/iommufd/main.c b/drivers/iommu/iommufd/main.c index 3fbe636c3d8a..32ce7befc8dd 100644 --- a/drivers/iommu/iommufd/main.c +++ b/drivers/iommu/iommufd/main.c @@ -50,7 +50,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx, * before calling iommufd_object_finalize(). */ rc = xa_alloc(&ictx->objects, &obj->id, XA_ZERO_ENTRY, - xa_limit_32b, GFP_KERNEL_ACCOUNT); + xa_limit_31b, GFP_KERNEL_ACCOUNT); if (rc) goto out_free; return obj; From patchwork Sat May 13 13:21:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D7A1C7EE23 for ; Sat, 13 May 2023 13:21:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238540AbjEMNVx (ORCPT ); Sat, 13 May 2023 09:21:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238270AbjEMNVt (ORCPT ); Sat, 13 May 2023 09:21:49 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B42A3C1E; Sat, 13 May 2023 06:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984107; x=1715520107; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=g4siJIBkFf/5Ervds9HnSwWatbMCuAZl2/EsF9oUoBE=; b=S1bFJOBGKG09KtOo/SyzxLShaLx0TNWcLzoW2ewTpKbD8AnLPp9oSEHH Mr3M2D0QKNDLG7xI9oVUeDRk4w7GbmK/68oZhlSZPgimXWxZVj2t/LsYv zcUX8n1nL20ZcUxwsxoMNQID+eZzev6GhMF6g1X5cB+CWLAWeZJBvwBs9 2OFUn5Soyrgv7eJv3Nzhltm2GV8h0wCiSlBSwwz1qul0QCLfTEq3V1X23 O2N7/mFVAse9aM5s0Kl6ElmKABNpxj5OrxCjFvGCOF6dDJw5me++iJbvk hEYfhR05o7OwLRBuw7Sjuczcj9WzuapliwWgg22/N76gZWHmU+ifdy1we w==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416599007" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416599007" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126460" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126460" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:46 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 06/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device Date: Sat, 13 May 2023 06:21:32 -0700 Message-Id: <20230513132136.15021-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is needed by the vfio-pci driver to report affected devices in the hot reset for a given device. Signed-off-by: Yi Liu --- drivers/iommu/iommufd/device.c | 24 ++++++++++++++++++++++++ drivers/vfio/iommufd.c | 20 ++++++++++++++++++++ include/linux/iommufd.h | 6 ++++++ include/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 64 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 4f9b2142274c..81466b97023f 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -116,6 +116,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + +u32 iommufd_device_to_id(struct iommufd_device *idev) +{ + return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) @@ -463,6 +475,18 @@ void iommufd_access_destroy(struct iommufd_access *access) } EXPORT_SYMBOL_NS_GPL(iommufd_access_destroy, IOMMUFD); +struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access) +{ + return access->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_access_to_ictx, IOMMUFD); + +u32 iommufd_access_to_id(struct iommufd_access *access) +{ + return access->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_access_to_id, IOMMUFD); + int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id) { struct iommufd_ioas *new_ioas; diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index c1379e826052..a18e920be164 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -105,6 +105,26 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (vdev->iommufd_device) + return iommufd_device_to_ictx(vdev->iommufd_device); + if (vdev->noiommu_access) + return iommufd_access_to_ictx(vdev->noiommu_access); + return NULL; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid(struct vfio_device *vdev) +{ + if (vdev->iommufd_device) + return iommufd_device_to_id(vdev->iommufd_device); + if (vdev->noiommu_access) + return iommufd_access_to_id(vdev->noiommu_access); + return -EINVAL; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid); + /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..68cd65274e28 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); +u32 iommufd_device_to_id(struct iommufd_device *idev); + struct iommufd_access_ops { u8 needs_pin_pages : 1; void (*unmap)(void *data, unsigned long iova, unsigned long length); @@ -45,6 +48,9 @@ iommufd_access_create(struct iommufd_ctx *ictx, void iommufd_access_destroy(struct iommufd_access *access); int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id); +struct iommufd_ctx *iommufd_access_to_ictx(struct iommufd_access *access); +u32 iommufd_access_to_id(struct iommufd_access *access); + void iommufd_ctx_get(struct iommufd_ctx *ictx); #if IS_ENABLED(CONFIG_IOMMUFD) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index a61130bc06a2..fcbe084b18c8 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -115,6 +115,8 @@ struct vfio_device_ops { }; #if IS_ENABLED(CONFIG_IOMMUFD) +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); +int vfio_iommufd_physical_devid(struct vfio_device *vdev); int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); @@ -124,6 +126,18 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev, void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); #else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + return NULL; +} + +static inline int +vfio_iommufd_physical_devid(struct vfio_device *vdev) +{ + return -EOPNOTSUPP; +} + #define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Sat May 13 13:21:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 187B3C77B7F for ; Sat, 13 May 2023 13:21:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238377AbjEMNVw (ORCPT ); Sat, 13 May 2023 09:21:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238057AbjEMNVt (ORCPT ); Sat, 13 May 2023 09:21:49 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9524C46A5; Sat, 13 May 2023 06:21:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984108; x=1715520108; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=L55vNyYMzhjBc2zuKvYV+2YX4T9Wqnkq3Ay94cJtU4M=; b=BeQ4IMvzm3we4aim/+O3K+EhOlazvP5Plx3qYFc+g7JfuRgQwaBKXSLH OnKPuoQODwCemV5UsGQhWL/uVHZ92GqmhHZbyfsO0xXE3mGD6CDZrmSUO QEugGvJn0VPqNVP14eEIY2h5i0F4UDFsU4/DxT55cjzywGqxVtmaubXTe ToAv4sCbFX6XZyiOvKsGLcgkIu5ZkKXGPzK6JNOUMSWu5DDUjDflfoaZy PBzu6LO/b/cj2OljmlewxiwessUywI5g0P33ouRLqn2ljunl+2ADvPIge P2kGiDgHYocTMDWPkFQQd4dvuXBaLfXbMhegVXWL+p3ApuyLrRkNouQRy g==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416599019" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416599019" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126464" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126464" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:47 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 07/10] vfio: Add helper to search vfio_device in a dev_set Date: Sat, 13 May 2023 06:21:33 -0700 Message-Id: <20230513132136.15021-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org There are drivers that need to search vfio_device within a given dev_set. e.g. vfio-pci. So add a helper. Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 8 +++----- drivers/vfio/vfio_main.c | 15 +++++++++++++++ include/linux/vfio.h | 3 +++ 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 39e7823088e7..4df2def35bdd 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2335,12 +2335,10 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, static int vfio_pci_is_device_in_set(struct pci_dev *pdev, void *data) { struct vfio_device_set *dev_set = data; - struct vfio_device *cur; - list_for_each_entry(cur, &dev_set->device_list, dev_set_list) - if (cur->dev == &pdev->dev) - return 0; - return -EBUSY; + lockdep_assert_held(&dev_set->lock); + + return vfio_find_device_in_devset(dev_set, &pdev->dev) ? 0 : -EBUSY; } /* diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index f0ca33b2e1df..ab4f3a794f78 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -141,6 +141,21 @@ unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set) } EXPORT_SYMBOL_GPL(vfio_device_set_open_count); +struct vfio_device * +vfio_find_device_in_devset(struct vfio_device_set *dev_set, + struct device *dev) +{ + struct vfio_device *cur; + + lockdep_assert_held(&dev_set->lock); + + list_for_each_entry(cur, &dev_set->device_list, dev_set_list) + if (cur->dev == dev) + return cur; + return NULL; +} +EXPORT_SYMBOL_GPL(vfio_find_device_in_devset); + /* * Device objects - create, release, get, put, search */ diff --git a/include/linux/vfio.h b/include/linux/vfio.h index fcbe084b18c8..4c17395ed4d2 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -259,6 +259,9 @@ void vfio_unregister_group_dev(struct vfio_device *device); int vfio_assign_device_set(struct vfio_device *device, void *set_id); unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set); +struct vfio_device * +vfio_find_device_in_devset(struct vfio_device_set *dev_set, + struct device *dev); int vfio_mig_get_next_state(struct vfio_device *device, enum vfio_device_mig_state cur_fsm, From patchwork Sat May 13 13:21:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76864C7EE23 for ; Sat, 13 May 2023 13:21:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238612AbjEMNV4 (ORCPT ); Sat, 13 May 2023 09:21:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238316AbjEMNVv (ORCPT ); Sat, 13 May 2023 09:21:51 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41D251FED; Sat, 13 May 2023 06:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984110; x=1715520110; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JPKn3Comiif0UnOhEse9L3Rl82bLgc6qyUnZga0nLkQ=; b=AV6IdomyiFr60Hr4OW9BxXUccV3qquY4ciLWVGHpGElb6DZhbI4WeuOP wNbRYJeHKPJSfyKr6AqDsZYJgkXjGC/plhjwgjH4Y3njoeI8dl9ffpsty vZIuSeaIMB2/Ol3Mz+0rw0dVd4mi+BRkBfSLvfhJT4NQn5J26SKR0XOTf f8zSykzMHnbHYE2YLP2imJzSwt3DE24Fg31iUNe4vMTt5SNOi6nzmcGdp jsmhWrrCYDBryMFskGbmMiYSgmzG7wiHmLKrwDE7LTrwCFDVNOnldu9kU 05Gp/rJEu+RkyANjkRpqIsIHSppHm5Z4n6rEB5G0WegAO7LuLlUme8tVD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416599028" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416599028" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126468" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126468" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:48 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 08/10] iommufd: Add iommufd_ctx_has_group() Date: Sat, 13 May 2023 06:21:34 -0700 Message-Id: <20230513132136.15021-9-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org to check if any device within the given iommu_group has been bound with the iommufd_ctx. This helpful for the checking on device ownership for the devices which have been bound but cannot be bound to any other iommufd as the iommu_group has been bound. Signed-off-by: Yi Liu --- drivers/iommu/iommufd/device.c | 29 +++++++++++++++++++++++++++++ include/linux/iommufd.h | 8 ++++++++ 2 files changed, 37 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 81466b97023f..5e5f7912807b 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -98,6 +98,35 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, } EXPORT_SYMBOL_NS_GPL(iommufd_device_bind, IOMMUFD); +/** + * iommufd_ctx_has_group - True if the struct device is bound to this ictx + * @ictx: iommufd file descriptor + * @group: Pointer to a physical iommu_group struct + * + * True if a iommufd_device_bind() is present for any device within the + * group. + */ +bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group) +{ + struct iommufd_object *obj; + unsigned long index; + + if (!ictx || !group) + return false; + + xa_lock(&ictx->objects); + xa_for_each(&ictx->objects, index, obj) { + if (obj->type == IOMMUFD_OBJ_DEVICE && + container_of(obj, struct iommufd_device, obj)->group == group) { + xa_unlock(&ictx->objects); + return true; + } + } + xa_unlock(&ictx->objects); + return false; +} +EXPORT_SYMBOL_NS_GPL(iommufd_ctx_has_group, IOMMUFD); + /** * iommufd_device_unbind - Undo iommufd_device_bind() * @idev: Device returned by iommufd_device_bind() diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 68cd65274e28..e49c16cd6831 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -16,6 +16,7 @@ struct page; struct iommufd_ctx; struct iommufd_access; struct file; +struct iommu_group; struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx, struct device *dev, u32 *id); @@ -56,6 +57,7 @@ void iommufd_ctx_get(struct iommufd_ctx *ictx); #if IS_ENABLED(CONFIG_IOMMUFD) struct iommufd_ctx *iommufd_ctx_from_file(struct file *file); void iommufd_ctx_put(struct iommufd_ctx *ictx); +bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group); int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, struct page **out_pages, @@ -77,6 +79,12 @@ static inline void iommufd_ctx_put(struct iommufd_ctx *ictx) { } +static inline bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, + struct iommu_group *group) +{ + return false; +} + static inline int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova, unsigned long length, From patchwork Sat May 13 13:21:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 521D5C77B7F for ; Sat, 13 May 2023 13:21:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238397AbjEMNV5 (ORCPT ); Sat, 13 May 2023 09:21:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238461AbjEMNVx (ORCPT ); Sat, 13 May 2023 09:21:53 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6F864204; Sat, 13 May 2023 06:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984111; x=1715520111; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WT52gKHGSak5vPDkA73jE4i14CaOOE70paj3Kw9bXfg=; b=bxV7urcQz1sobjt9oS5MzkW/BbZ2HvVn9mqwQLQvu+ZrW5MROYKggs4o hgYf1K6oWvo1Nj9nH7Fg5++3UF/i6Azn5MZQExxg7IDnlgYzyQsSEKlgw OUq3gMRDYGwOKCvg5FPWtrDbHXakX040+EiIC3k7WLywi4Q9gXeOB2Ie5 1SNjYA1EX584dzyZlhCBHGfyu3Y1YkNqEk3vzqzm0xfh9n2ns+d4JE4Hr Gc0gOKWhi/nfYft/vswKFkmG+csLlVkEBqLSQffZnr5R7deC2rE1gE6wL VZ1t/VA7t8G/iXC0Ns4rEQ0ftWsnBLJkaNrHrkHQcyC8hOlm/JZo45Hof Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416599038" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416599038" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126478" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126478" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:50 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 09/10] vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev Date: Sat, 13 May 2023 06:21:35 -0700 Message-Id: <20230513132136.15021-10-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This makes VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl to use the iommufd_ctx of the cdev device to check the ownership of the other affected devices. This returns devid for each of the affected devices. If it is bound to the iommufd_ctx of the cdev device, _INFO reports a valid devid > 0; If it is not opened by the calling user, but it belongs to the same iommu_group of a device that is bound to the iommufd_ctx of the cdev device, reports devid value of 0; If the device is un-owned device, configured within a different iommufd, or opened outside of the vfio device cdev API, the _INFO ioctl shall report devid value of -1. devid >=0 doesn't block hot-reset as the affected devices are considered to be owned, while devid == -1 will block the use of VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling conventions (ie. via legacy group accessed devices). This adds flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID to tell the user devid is returned in case of calling user get device fd from other software stack and adds flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED to tell user if all the affected devices are owned, so user can know it without looping all the returned devids. Suggested-by: Jason Gunthorpe Suggested-by: Alex Williamson Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++++++++++++++++-- include/uapi/linux/vfio.h | 46 +++++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 4df2def35bdd..57586be770af 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -27,6 +27,7 @@ #include #include #include +#include #if IS_ENABLED(CONFIG_EEH) #include #endif @@ -36,6 +37,10 @@ #define DRIVER_AUTHOR "Alex Williamson " #define DRIVER_DESC "core driver for VFIO based PCI devices" +#ifdef CONFIG_IOMMUFD +MODULE_IMPORT_NS(IOMMUFD); +#endif + static bool nointxmask; static bool disable_vga; static bool disable_idle_d3; @@ -776,6 +781,9 @@ struct vfio_pci_fill_info { int max; int cur; struct vfio_pci_dependent_device *devices; + struct vfio_device *vdev; + bool devid:1; + bool dev_owned:1; }; static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) @@ -790,7 +798,37 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) if (!iommu_group) return -EPERM; /* Cannot reset non-isolated devices */ - fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + if (fill->devid) { + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(fill->vdev); + struct vfio_device_set *dev_set = fill->vdev->dev_set; + struct vfio_device *vdev; + + /* + * Report devid for the affected devices: + * - valid devid > 0 for the devices that are bound with + * the iommufd of the calling device. + * - devid == 0 for the devices that have not been opened + * but have same group with one of the devices bound to + * the iommufd of the calling device. + * - devid == -1 for others, and clear dev_owned flag. + */ + vdev = vfio_find_device_in_devset(dev_set, &pdev->dev); + if (vdev && iommufd == vfio_iommufd_physical_ictx(vdev)) { + int ret; + + ret = vfio_iommufd_physical_devid(vdev); + if (WARN_ON(ret < 0)) + return ret; + fill->devices[fill->cur].devid = ret; + } else if (vdev && iommufd_ctx_has_group(iommufd, iommu_group)) { + fill->devices[fill->cur].devid = VFIO_PCI_DEVID_OWNED; + } else { + fill->devices[fill->cur].devid = VFIO_PCI_DEVID_NOT_OWNED; + fill->dev_owned = false; + } + } else { + fill->devices[fill->cur].group_id = iommu_group_id(iommu_group); + } fill->devices[fill->cur].segment = pci_domain_nr(pdev->bus); fill->devices[fill->cur].bus = pdev->bus->number; fill->devices[fill->cur].devfn = pdev->devfn; @@ -1229,17 +1267,27 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return -ENOMEM; fill.devices = devices; + fill.vdev = &vdev->vdev; + mutex_lock(&vdev->vdev.dev_set->lock); + fill.devid = fill.dev_owned = vfio_device_cdev_opened(&vdev->vdev); ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_fill_devs, &fill, slot); + mutex_unlock(&vdev->vdev.dev_set->lock); /* * If a device was removed between counting and filling, we may come up * short of fill.max. If a device was added, we'll have a return of * -EAGAIN above. */ - if (!ret) + if (!ret) { hdr.count = fill.cur; + if (fill.devid) { + hdr.flags |= VFIO_PCI_HOT_RESET_FLAG_DEV_ID; + if (fill.dev_owned) + hdr.flags |= VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED; + } + } reset_info_exit: if (copy_to_user(arg, &hdr, minsz)) diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..01203215251a 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -650,11 +650,53 @@ enum { * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, * struct vfio_pci_hot_reset_info) * + * This command is used to query the affected devices in the hot reset for + * a given device. + * + * This command always reports the segment, bus, and devfn information for + * each affected device, and selectively reports the group_id or devid per + * the way how the calling device is opened. + * + * - If the calling device is opened via the traditional group/container + * API, group_id is reported. User should check if it has owned all + * the affected devices and provides a set of group fds to prove the + * ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl. + * + * - If the calling device is opened as a cdev, devid is reported. + * Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this + * data type. For a given affected device, it is considered owned by + * this interface if it meets the following conditions: + * 1) Has a valid devid within the iommufd_ctx of the calling device. + * Ownership cannot be determined across separate iommufd_ctx and the + * cdev calling conventions do not support a proof-of-ownership model + * as provided in the legacy group interface. In this case a valid + * devid with value greater than zero is provided in the return + * structure. + * 2) Does not have a valid devid within the iommufd_ctx of the calling + * device, but belongs to the same IOMMU group as the calling device + * or another opened device that has a valid devid within the + * iommufd_ctx of the calling device. This provides implicit ownership + * for devices within the same DMA isolation context. In this case + * the invalid devid value of zero is provided in the return structure. + * + * A devid value of -1 is provided in the return structure for devices + * where ownership is not available. Such devices prevent the use of + * VFIO_DEVICE_PCI_HOT_RESET outside of proof-of-ownership calling + * conventions (ie. via legacy group accessed devices). + * Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the + * affected devices are owned by the user. This flag is available only + * when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. + * * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. */ struct vfio_pci_dependent_device { - __u32 group_id; + union { + __u32 group_id; + __u32 devid; +#define VFIO_PCI_DEVID_OWNED 0 +#define VFIO_PCI_DEVID_NOT_OWNED -1 + }; __u16 segment; __u8 bus; __u8 devfn; /* Use PCI_SLOT/PCI_FUNC */ @@ -663,6 +705,8 @@ struct vfio_pci_dependent_device { struct vfio_pci_hot_reset_info { __u32 argsz; __u32 flags; +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0) +#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1) __u32 count; struct vfio_pci_dependent_device devices[]; }; From patchwork Sat May 13 13:21:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13240177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAC62C77B7D for ; Sat, 13 May 2023 13:22:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238704AbjEMNWA (ORCPT ); Sat, 13 May 2023 09:22:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238565AbjEMNVy (ORCPT ); Sat, 13 May 2023 09:21:54 -0400 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 954AD46A9; Sat, 13 May 2023 06:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683984112; x=1715520112; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lEhjBYfia0mz7JSmjh7SFm1OSylP/YfJVuAcamAGQhs=; b=OOz3BUzhiTLwyzBan0/XkUtOL4bcLOie7BVoOL1FsbOtzHmUFXwFEGsU SV1O1tQY72JNDP0IeNfDDStF2NGLnSn4MyWn9jVE0r3aP6xR1FNUBIJer Q5WRIMiqLz+AlTDBO8ga1sRLGKOjMqJLQsfxXGfRet7qnC5RKhllv9jZm 3HioHFWKYujVXOrPpuc/v3OCqNAxm/Y0jleUHspee3lPXA1qKQbIJXhtu m7UpWy+wjoPpQSQRJ2uCIZex2sciQS+cnns7NHuYXLIpxj9RcqE8vwiPK MBR5s5AUtbIQYx3nC4vJ0ckmwxZrxOLMwRrFB9Ufj/EKEaqMvz15j/KH4 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="416599051" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="416599051" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 May 2023 06:21:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10708"; a="790126486" X-IronPort-AV: E=Sophos;i="5.99,272,1677571200"; d="scan'208";a="790126486" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by FMSMGA003.fm.intel.com with ESMTP; 13 May 2023 06:21:51 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com, zhenzhong.duan@intel.com, clegoate@redhat.com Subject: [PATCH v5 10/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET Date: Sat, 13 May 2023 06:21:36 -0700 Message-Id: <20230513132136.15021-11-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230513132136.15021-1-yi.l.liu@intel.com> References: <20230513132136.15021-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is the way user to invoke hot-reset for the devices opened by cdev interface. User should check the flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED in the output of VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl before doing hot-reset for cdev devices. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe Tested-by: Yanting Jiang Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 68 ++++++++++++++++++++++++++------ include/uapi/linux/vfio.h | 14 +++++++ 2 files changed, 71 insertions(+), 11 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 57586be770af..7fa3f54eb855 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -185,7 +185,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1323,8 +1324,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (ret) return ret; - /* Somewhere between 1 and count is OK */ - if (!array_count || array_count > count) + if (array_count > count || vfio_device_cdev_opened(&vdev->vdev)) return -EINVAL; group_fds = kcalloc(array_count, sizeof(*group_fds), GFP_KERNEL); @@ -1373,7 +1373,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = array_count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1402,7 +1402,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, else if (pci_probe_reset_bus(vdev->pdev->bus)) return -ENODEV; - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, hdr.count, slot, arg); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, + vfio_iommufd_physical_ictx(&vdev->vdev)); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, @@ -2369,13 +2373,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { }; EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, +static bool vfio_dev_in_groups(struct vfio_device *vdev, struct vfio_pci_group_info *groups) { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) - if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) + if (vfio_file_has_dev(groups->files[i], vdev)) return true; return false; } @@ -2447,13 +2454,37 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_device_owned(struct vfio_device *vdev, + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommu_group *group; + + WARN_ON(!!groups == !!iommufd_ctx); + + if (groups) + return vfio_dev_in_groups(vdev, groups); + + if (vfio_iommufd_physical_ictx(vdev) == iommufd_ctx) + return true; + + group = iommu_group_get(vdev->dev); + if (!group) + return false; + + iommu_group_put(group); + + return iommufd_ctx_has_group(iommufd_ctx, group); +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2484,10 +2515,25 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * If the user provides a set of groups, all the devices + * in the dev_set should be contained by the set of groups + * provided by the user. + * + * If the user provides a zero-length group fd array, then + * all the devices in the dev_set must be bound to the same + * iommufd_ctx as the input iommufd_ctx. If there is any + * device that has not been bound to iommufd_ctx yet, check + * if its iommu_group has any device bound to the input + * iommufd_ctx Such devices can be considered owned by + * the input iommufd_ctx as the device cannot be owned + * by another iommufd_ctx when its iommu_group is owned. + * + * Otherwise, reset is not allowed. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (!vfio_device_owned(&cur_vma->vdev, groups, iommufd_ctx)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 01203215251a..24858b650562 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -686,6 +686,9 @@ enum { * Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the * affected devices are owned by the user. This flag is available only * when VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved. + * When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero + * length fd array on the calling device as the ownership is validated + * by iommufd_ctx. * * Return: 0 on success, -errno on failure: * -enospc = insufficient buffer, -enodev = unsupported for device. @@ -717,6 +720,17 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it operates. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices are owned by itself. + * + * As the ownership described by VFIO_DEVICE_GET_PCI_HOT_RESET_INFO, the + * cdev opened devices must exclusively provide a zero-length fd array and + * the group opened devices must exclusively use an array of group fds for + * proof of ownership. Mixed access to devices between cdev and legacy + * groups are not supported by this interface. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset {