From patchwork Mon Mar 27 09:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F282C761AF for ; Mon, 27 Mar 2023 09:35:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233378AbjC0Jft (ORCPT ); Mon, 27 Mar 2023 05:35:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233280AbjC0Jfd (ORCPT ); Mon, 27 Mar 2023 05:35:33 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F20955AE; Mon, 27 Mar 2023 02:35:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909730; x=1711445730; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+VKOyERweJfYAdfXicxL5YrmE4b4662Jc+V+KNNxwM8=; b=FyDLlzyaIMFUkwAF8EaoPE/IRzTo6p3B26N4+JQymUKX0gVvDqxF+SZh v8cWedPGp2VyBWLRdxgzn/gcx/Fm1WfCsRvoDVGalTUvi8OBKr5LyCvPT TCPmcPKODy7WEkdIafXmWqzvOADJ1aY0EPFDfOd8Waq3xI+tVdpdWIkaP crk4RnDLGqYQNQ3+oR8UQWm0UpIyq2AZ3F9CD0zj5AZnK61B2zYqT/uW0 0dUnZjz84enmrZoYeOsdIfF+lIoxaIHx/v15QpmPDR4izkwwbkueF27UU y0gxqxjQan61s7+h2HKBirXgaTLx8jRHPoV1uapk3GToWejHb8L3iD9qn w==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879485" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879485" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554605" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554605" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:00 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 01/10] vfio/pci: Update comment around group_fd get in vfio_pci_ioctl_pci_hot_reset() Date: Mon, 27 Mar 2023 02:34:49 -0700 Message-Id: <20230327093458.44939-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org this suits more on what the code does. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index a5ab416cf476..65bbef562268 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,9 +1308,8 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } /* - * For each group_fd, get the group through the vfio external user - * interface and store the group and iommu ID. This ensures the group - * is held across the reset. + * Get the group file for each fd to ensure the group held across + * the reset */ for (file_idx = 0; file_idx < hdr.count; file_idx++) { struct file *file = fget(group_fds[file_idx]); From patchwork Mon Mar 27 09:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E1DFC77B62 for ; Mon, 27 Mar 2023 09:35:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233393AbjC0Jfu (ORCPT ); Mon, 27 Mar 2023 05:35:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233392AbjC0Jfe (ORCPT ); Mon, 27 Mar 2023 05:35:34 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D019A55A5; Mon, 27 Mar 2023 02:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909732; x=1711445732; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=d7FNJi55AD86wuPAjqm6ryTv0JA15p82567AVXSPbzY=; b=bwqINojAPBdXkMRbfzIBZpIpOtXui7aYd+wakpdkioW24HYvq5rjihfD P1XdKsjQuMc34aOmXOAC2yScK9RKOA6e+2ZvdcRPo+1YVOggpglP+ks8O QVJTVIjJfYIbfbTK0L6LjgThddFQLfcZR1zAPallWJoRACS7rhHHkYAAp AZ1OftxCzUQpEGwm8rmiq+WwyoOAtjS9FV2rk1bxNrtKb+t1BGZiQuijl 579NnGEpQtAz9AM2xhSeoda9R/5ys6greHWrvRzghwApjc+gPAKc38Y+z 30Z4zulH5Grc1BGQaiTkSJi21v8d/j17UGKjHlkfTOfFmqbxao0H4MGRK Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879498" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879498" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554616" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554616" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:01 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 02/10] vfio/pci: Only check ownership of opened devices in hot reset Date: Mon, 27 Mar 2023 02:34:50 -0700 Message-Id: <20230327093458.44939-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If the affected device is not opened by any user, it's safe to reset it given it's not in use. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 14 +++++++++++--- include/uapi/linux/vfio.h | 8 ++++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 65bbef562268..5d745c9abf05 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -2429,10 +2429,18 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, list_for_each_entry(cur_vma, &dev_set->device_list, vdev.dev_set_list) { /* - * Test whether all the affected devices are contained by the - * set of groups provided by the user. + * Test whether all the affected devices can be reset by the + * user. + * + * Resetting an unused device (not opened) is safe, because + * dev_set->lock is held in hot reset path so this device + * cannot race being opened by another user simultaneously. + * + * Otherwise all opened devices in the dev_set must be + * contained by the set of groups provided by the user. */ - if (!vfio_dev_in_groups(cur_vma, groups)) { + if (cur_vma->vdev.open_count && + !vfio_dev_in_groups(cur_vma, groups)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 0552e8dcf0cb..f96e5689cffc 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -673,6 +673,14 @@ struct vfio_pci_hot_reset_info { * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset) * + * Userspace requests hot reset for the devices it uses. Due to the + * underlying topology, multiple devices can be affected in the reset + * while some might be opened by another user. To avoid interference + * the calling user must ensure all affected devices, if opened, are + * owned by itself. + * + * The ownership is proved by an array of group fds. + * * Return: 0 on success, -errno on failure. */ struct vfio_pci_hot_reset { From patchwork Mon Mar 27 09:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3619BC77B62 for ; Mon, 27 Mar 2023 09:35:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233259AbjC0Jfx (ORCPT ); Mon, 27 Mar 2023 05:35:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233502AbjC0Jff (ORCPT ); Mon, 27 Mar 2023 05:35:35 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64E8249E2; Mon, 27 Mar 2023 02:35:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909733; x=1711445733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jbMe+F9CADAi8l9/BVZxSAdSYRuI/B7FG31MAnL6+cI=; b=AqGuCQUFlpsDMZtlkEboO0ZMAfiWFluhLMNrFPB/iKPXeBN53GnecqWE c1JnQ25fz3w6uPzOQAEEA0d5UV7f6mab/jRSLard1Mwln3dtjQAIthZMu fCyuspvLdqY97XFwKNwiuILBn/apZwwd1GUQMRuzTTiUvns0GMAIcu5Kx st0zZkH9PbtTQwMTUKq89nlyZEiN2N7m+CraxeDb1jp6s3y7cWE9eWT4E 2DQUjccKT3JhAGkH0q03alkCvymnKg9fVsI/6YZ1are3tOe3YLZolWYAN CmZM7Iow5yhr8ks274S8jFnClyoovSf++X1Il3SpREsMsv9hizou8ggRO A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879507" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879507" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554626" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554626" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:02 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 03/10] vfio/pci: Move the existing hot reset logic to be a helper Date: Mon, 27 Mar 2023 02:34:51 -0700 Message-Id: <20230327093458.44939-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This prepares to add another method for hot reset. The major hot reset logic are moved to vfio_pci_ioctl_pci_hot_reset_groups(). No functional change is intended. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 56 +++++++++++++++++++------------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 5d745c9abf05..3696b8e58445 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1255,29 +1255,17 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( return ret; } -static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset __user *arg) +static int +vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); - struct vfio_pci_hot_reset hdr; int32_t *group_fds; struct file **files; struct vfio_pci_group_info info; - bool slot = false; int file_idx, count = 0, ret = 0; - if (copy_from_user(&hdr, arg, minsz)) - return -EFAULT; - - if (hdr.argsz < minsz || hdr.flags) - return -EINVAL; - - /* Can we do a slot or bus reset or neither? */ - if (!pci_probe_reset_slot(vdev->pdev->slot)) - slot = true; - else if (pci_probe_reset_bus(vdev->pdev->bus)) - return -ENODEV; - /* * We can't let userspace give us an arbitrarily large buffer to copy, * so verify how many we think there could be. Note groups can have @@ -1289,11 +1277,11 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr.count || hdr.count > count) + if (!hdr->count || hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr.count, sizeof(*group_fds), GFP_KERNEL); - files = kcalloc(hdr.count, sizeof(*files), GFP_KERNEL); + group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); if (!group_fds || !files) { kfree(group_fds); kfree(files); @@ -1301,7 +1289,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, } if (copy_from_user(group_fds, arg->group_fds, - hdr.count * sizeof(*group_fds))) { + hdr->count * sizeof(*group_fds))) { kfree(group_fds); kfree(files); return -EFAULT; @@ -1311,7 +1299,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, * Get the group file for each fd to ensure the group held across * the reset */ - for (file_idx = 0; file_idx < hdr.count; file_idx++) { + for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(group_fds[file_idx]); if (!file) { @@ -1335,7 +1323,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, if (ret) goto hot_reset_release; - info.count = hdr.count; + info.count = hdr->count; info.files = files; ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); @@ -1348,6 +1336,28 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return ret; } +static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset __user *arg) +{ + unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); + struct vfio_pci_hot_reset hdr; + bool slot = false; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz || hdr.flags) + return -EINVAL; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); +} + static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, struct vfio_device_ioeventfd __user *arg) { From patchwork Mon Mar 27 09:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38078C761A6 for ; Mon, 27 Mar 2023 09:35:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233414AbjC0Jfv (ORCPT ); Mon, 27 Mar 2023 05:35:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233403AbjC0Jff (ORCPT ); Mon, 27 Mar 2023 05:35:35 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64F665258; Mon, 27 Mar 2023 02:35:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909733; x=1711445733; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GInm+3DnElLmSvJJLYAMBhnzd2opXH4SR3j570LhFkU=; b=YSXQAbc7ONlWkk1M3nC6I5xCvVK8voqgMMR0mG3Nf+P+rIghgBatM3uy FkYIEIecqFkVfn3WGJdd9TddIsIEvfMXaDVzbe1E9EvpcC+UdJSozlS6e elVGrnhOmUwJecJuJexNvguqcnI5reYoqLuOAYo5THPkOcSaRTzv64slw KXk10CqhLWc9+ibkcQJX7/6XdPpbe1Xku9NUMAxvRNLlV6WnoFaozjtVm 2t2iPy452HExH+E7uM7DHpWRQEwqo7ASifzmOTRMCQwp23ASoGLKMSMa2 60Zs53Rqjm8qtJCTOkk0hwT47q72hAzwmtploc8whHblxGwBrAXNKAjHG Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879519" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879519" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554634" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554634" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:03 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 04/10] vfio-iommufd: Add helper to retrieve iommufd_ctx and devid for vfio_device Date: Mon, 27 Mar 2023 02:34:52 -0700 Message-Id: <20230327093458.44939-5-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is needed by the vfio pci driver to report affected devices in the hot reset for a given device. Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/iommu/iommufd/device.c | 12 ++++++++++++ drivers/vfio/iommufd.c | 16 ++++++++++++++++ include/linux/iommufd.h | 3 +++ include/linux/vfio.h | 13 +++++++++++++ 4 files changed, 44 insertions(+) diff --git a/drivers/iommu/iommufd/device.c b/drivers/iommu/iommufd/device.c index 25115d401d8f..04a57aa1ae2c 100644 --- a/drivers/iommu/iommufd/device.c +++ b/drivers/iommu/iommufd/device.c @@ -131,6 +131,18 @@ void iommufd_device_unbind(struct iommufd_device *idev) } EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev) +{ + return idev->ictx; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD); + +u32 iommufd_device_to_id(struct iommufd_device *idev) +{ + return idev->obj.id; +} +EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD); + static int iommufd_device_setup_msi(struct iommufd_device *idev, struct iommufd_hw_pagetable *hwpt, phys_addr_t sw_msi_start) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 88b00c501015..44088049dbb1 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -66,6 +66,22 @@ void vfio_iommufd_unbind(struct vfio_device *vdev) vdev->ops->unbind_iommufd(vdev); } +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + if (!vdev->iommufd_device) + return NULL; + return iommufd_device_to_ictx(vdev->iommufd_device); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_ictx); + +int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ + if (!vdev->iommufd_device) + return -EINVAL; + *id = iommufd_device_to_id(vdev->iommufd_device); + return 0; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_devid); /* * The physical standard ops mean that the iommufd_device is bound to the * physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers diff --git a/include/linux/iommufd.h b/include/linux/iommufd.h index 1129a36a74c4..ac96df406833 100644 --- a/include/linux/iommufd.h +++ b/include/linux/iommufd.h @@ -24,6 +24,9 @@ void iommufd_device_unbind(struct iommufd_device *idev); int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id); void iommufd_device_detach(struct iommufd_device *idev); +struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev); +u32 iommufd_device_to_id(struct iommufd_device *idev); + struct iommufd_access_ops { u8 needs_pin_pages : 1; void (*unmap)(void *data, unsigned long iova, unsigned long length); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 3188d8a374bd..e54bef5489a0 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -113,6 +113,8 @@ struct vfio_device_ops { }; #if IS_ENABLED(CONFIG_IOMMUFD) +struct iommufd_ctx *vfio_iommufd_physical_ictx(struct vfio_device *vdev); +int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id); int vfio_iommufd_physical_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_physical_unbind(struct vfio_device *vdev); @@ -122,6 +124,17 @@ int vfio_iommufd_emulated_bind(struct vfio_device *vdev, void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id); #else +static inline struct iommufd_ctx * +vfio_iommufd_physical_ictx(struct vfio_device *vdev) +{ + return NULL; +} + +static inline int vfio_iommufd_physical_devid(struct vfio_device *vdev, u32 *id) +{ + return -EOPNOTSUPP; +} + #define vfio_iommufd_physical_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Mon Mar 27 09:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D729FC761A6 for ; Mon, 27 Mar 2023 09:35:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232453AbjC0Jfz (ORCPT ); Mon, 27 Mar 2023 05:35:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233490AbjC0Jfh (ORCPT ); Mon, 27 Mar 2023 05:35:37 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D296735A6; Mon, 27 Mar 2023 02:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909734; x=1711445734; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DspUHqM3GniNFDpFuKlLGJpDgU4kjJg6COLxwo46xbI=; b=cx7pVr03b5BujxX5ozpzhfBusBlzQLfBUB21VU+zDtPDIY0YkdfaT8Al HOLNldqbzcaDeES3eyyIkeLmalTTv0sfoerDSCmq3awHFz8Pj4kv+WOh8 29jcsgaNmYEkcnuvcE0SVnF/VVfKF1S1kaI2xeKrCwBo0YUsuv7wEYLU3 xk739PYdEIlSpWcKX0TFl1k9xrXV9Fnw5Qx9V4OTEOVawEetRaUo2Gbhs QLTGq6LHfXOzQCRDDb2saov8YXhpadGKwAQu07GDwf8bL9ceayk2DCwJG Yx+l+MAaHiSlavQMp8ZCIDwSGZ/40gGdKeUBOgBBeOU2BE7ftBnnLdGHg w==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879532" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879532" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554643" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554643" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:04 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 05/10] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET Date: Mon, 27 Mar 2023 02:34:53 -0700 Message-Id: <20230327093458.44939-6-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org as an alternative method for ownership check when iommufd is used. In this case all opened devices in the affected dev_set are verified to be bound to a same valid iommufd value to allow reset. It's simpler and faster as user does not need to pass a set of fds and kernel no need to search the device within the given fds. a device in noiommu mode doesn't have a valid iommufd, so this method should not be used in a dev_set which contains multiple devices and one of them is in noiommu. The only allowed noiommu scenario is that the calling device is noiommu and it's in a singleton dev_set. Suggested-by: Jason Gunthorpe Signed-off-by: Jason Gunthorpe Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 42 +++++++++++++++++++++++++++----- include/uapi/linux/vfio.h | 9 ++++++- 2 files changed, 44 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 3696b8e58445..b68fcba67a4b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -180,7 +180,8 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) struct vfio_pci_group_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups); + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx); /* * INTx masking requires the ability to disable INTx signaling via PCI_COMMAND @@ -1277,7 +1278,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, return ret; /* Somewhere between 1 and count is OK */ - if (!hdr->count || hdr->count > count) + if (hdr->count > count) return -EINVAL; group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); @@ -1326,7 +1327,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, info.count = hdr->count; info.files = files; - ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info); + ret = vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, &info, NULL); hot_reset_release: for (file_idx--; file_idx >= 0; file_idx--) @@ -1341,6 +1342,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, { unsigned long minsz = offsetofend(struct vfio_pci_hot_reset, count); struct vfio_pci_hot_reset hdr; + struct iommufd_ctx *iommufd; bool slot = false; if (copy_from_user(&hdr, arg, minsz)) @@ -1355,7 +1357,12 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, else if (pci_probe_reset_bus(vdev->pdev->bus)) return -ENODEV; - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + if (hdr.count) + return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + return vfio_pci_dev_set_hot_reset(vdev->vdev.dev_set, NULL, iommufd); } static int vfio_pci_ioctl_ioeventfd(struct vfio_pci_core_device *vdev, @@ -2327,6 +2334,9 @@ static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, { unsigned int i; + if (!groups) + return false; + for (i = 0; i < groups->count; i++) if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) return true; @@ -2402,13 +2412,25 @@ static int vfio_pci_dev_set_pm_runtime_get(struct vfio_device_set *dev_set) return ret; } +static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, + struct iommufd_ctx *iommufd_ctx) +{ + struct iommufd_ctx *iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + + if (!iommufd) + return false; + + return iommufd == iommufd_ctx; +} + /* * We need to get memory_lock for each device, but devices can share mmap_lock, * therefore we need to zap and hold the vma_lock for each device, and only then * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups) + struct vfio_pci_group_info *groups, + struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; struct vfio_pci_core_device *cur_vma; @@ -2448,9 +2470,17 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * * Otherwise all opened devices in the dev_set must be * contained by the set of groups provided by the user. + * + * If user provides a zero-length array, then all the + * opened devices must be bound to a same iommufd_ctx. + * + * If all above checks are failed, reset is allowed only if + * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups)) { + !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && + (dev_set->device_count > 1)) { ret = -EINVAL; goto err_undo; } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index f96e5689cffc..17aa5d09db41 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -679,7 +679,14 @@ struct vfio_pci_hot_reset_info { * the calling user must ensure all affected devices, if opened, are * owned by itself. * - * The ownership is proved by an array of group fds. + * The ownership can be proved by: + * - An array of group fds + * - A zero-length array + * + * In the last case all affected devices which are opened by this user + * must have been bound to a same iommufd. If the calling device is in + * noiommu mode (no valid iommufd) then it can be reset only if the reset + * doesn't affect other devices. * * Return: 0 on success, -errno on failure. */ From patchwork Mon Mar 27 09:34:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8034CC76195 for ; Mon, 27 Mar 2023 09:35:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233358AbjC0Jf5 (ORCPT ); Mon, 27 Mar 2023 05:35:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233508AbjC0Jfh (ORCPT ); Mon, 27 Mar 2023 05:35:37 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59305524C; Mon, 27 Mar 2023 02:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909735; x=1711445735; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lPjbsC97jhsaoOPvq0BOI8tQ/LPFyBBuvflvbc0YitU=; b=Xj4eQccL/qb10Rc3vFGqLyxejzbHa5gmOnhU1ZqnfTF/c6bWhUcNNToO Mwv7p4YfPTOwljLyCbOxDhnEoCBYCDfyLHfJLueV+DDeyys7HMgOFGuS/ +LOW1eJOSfqLnZ6E+VyZrSkrslfQxeQzf0RWqx6IYlgljXyKC9DUHd/hZ OTLXSBRcYZrtNd9Dj2lmxctMWfBVWyKw6ns/T6IbHbDzjPJXqT0fh/elA dhC9ySZnDFQb2pVpXW4VJi6TH6W25jPJYhwV5Zceb/1MNL4T62qiLPw6+ Xm9/dx7g3B2l+qsu0q8McAjNZujNkMiSExQ77jEQsEMKwFBAgwW6d8JmH Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879542" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879542" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554649" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554649" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:05 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 06/10] vfio: Refine vfio file kAPIs for vfio PCI hot reset Date: Mon, 27 Mar 2023 02:34:54 -0700 Message-Id: <20230327093458.44939-7-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This prepares vfio core to accept vfio device file from the vfio PCI hot reset path. vfio_file_is_group() is still kept for KVM usage. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/group.c | 32 ++++++++++++++------------------ drivers/vfio/pci/vfio_pci_core.c | 4 ++-- drivers/vfio/vfio.h | 2 ++ drivers/vfio/vfio_main.c | 29 +++++++++++++++++++++++++++++ include/linux/vfio.h | 1 + 5 files changed, 48 insertions(+), 20 deletions(-) diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c index 27d5ba7cf9dc..d0c95d033605 100644 --- a/drivers/vfio/group.c +++ b/drivers/vfio/group.c @@ -745,6 +745,15 @@ bool vfio_device_has_container(struct vfio_device *device) return device->group->container; } +struct vfio_group *vfio_group_from_file(struct file *file) +{ + struct vfio_group *group = file->private_data; + + if (file->f_op != &vfio_group_fops) + return NULL; + return group; +} + /** * vfio_file_iommu_group - Return the struct iommu_group for the vfio group file * @file: VFIO group file @@ -755,13 +764,13 @@ bool vfio_device_has_container(struct vfio_device *device) */ struct iommu_group *vfio_file_iommu_group(struct file *file) { - struct vfio_group *group = file->private_data; + struct vfio_group *group = vfio_group_from_file(file); struct iommu_group *iommu_group = NULL; if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU)) return NULL; - if (!vfio_file_is_group(file)) + if (!group) return NULL; mutex_lock(&group->group_lock); @@ -775,12 +784,12 @@ struct iommu_group *vfio_file_iommu_group(struct file *file) EXPORT_SYMBOL_GPL(vfio_file_iommu_group); /** - * vfio_file_is_group - True if the file is usable with VFIO aPIS + * vfio_file_is_group - True if the file is a vfio group file * @file: VFIO group file */ bool vfio_file_is_group(struct file *file) { - return file->f_op == &vfio_group_fops; + return vfio_group_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_group); @@ -842,23 +851,10 @@ void vfio_file_set_kvm(struct file *file, struct kvm *kvm) } EXPORT_SYMBOL_GPL(vfio_file_set_kvm); -/** - * vfio_file_has_dev - True if the VFIO file is a handle for device - * @file: VFIO file to check - * @device: Device that must be part of the file - * - * Returns true if given file has permission to manipulate the given device. - */ -bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device) { - struct vfio_group *group = file->private_data; - - if (!vfio_file_is_group(file)) - return false; - return group == device->group; } -EXPORT_SYMBOL_GPL(vfio_file_has_dev); static char *vfio_devnode(const struct device *dev, umode_t *mode) { diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index b68fcba67a4b..2a510b71edcb 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1308,8 +1308,8 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, break; } - /* Ensure the FD is a vfio group FD.*/ - if (!vfio_file_is_group(file)) { + /* Ensure the FD is a vfio FD. vfio group or vfio device */ + if (!vfio_file_is_valid(file)) { fput(file); ret = -EINVAL; break; diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 7b19c621e0e6..c0aeea24fbd6 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -84,6 +84,8 @@ void vfio_device_group_unregister(struct vfio_device *device); int vfio_device_group_use_iommu(struct vfio_device *device); void vfio_device_group_unuse_iommu(struct vfio_device *device); void vfio_device_group_close(struct vfio_device *device); +struct vfio_group *vfio_group_from_file(struct file *file); +bool vfio_group_has_dev(struct vfio_group *group, struct vfio_device *device); bool vfio_device_has_container(struct vfio_device *device); int __init vfio_group_init(void); void vfio_group_cleanup(void); diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 89497c933490..fe7446805afd 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,6 +1154,35 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +/** + * vfio_file_is_valid - True if the file is valid vfio file + * @file: VFIO group file or VFIO device file + */ +bool vfio_file_is_valid(struct file *file) +{ + return vfio_group_from_file(file); +} +EXPORT_SYMBOL_GPL(vfio_file_is_valid); + +/** + * vfio_file_has_dev - True if the VFIO file is a handle for device + * @file: VFIO file to check + * @device: Device that must be part of the file + * + * Returns true if given file has permission to manipulate the given device. + */ +bool vfio_file_has_dev(struct file *file, struct vfio_device *device) +{ + struct vfio_group *group; + + group = vfio_group_from_file(file); + if (!group) + return false; + + return vfio_group_has_dev(group, device); +} +EXPORT_SYMBOL_GPL(vfio_file_has_dev); + /* * Sub-module support */ diff --git a/include/linux/vfio.h b/include/linux/vfio.h index e54bef5489a0..79c47733ae0d 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -258,6 +258,7 @@ int vfio_mig_get_next_state(struct vfio_device *device, */ struct iommu_group *vfio_file_iommu_group(struct file *file); bool vfio_file_is_group(struct file *file); +bool vfio_file_is_valid(struct file *file); bool vfio_file_enforced_coherent(struct file *file); void vfio_file_set_kvm(struct file *file, struct kvm *kvm); bool vfio_file_has_dev(struct file *file, struct vfio_device *device); From patchwork Mon Mar 27 09:34:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3F5BC761A6 for ; Mon, 27 Mar 2023 09:35:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233301AbjC0Jf6 (ORCPT ); Mon, 27 Mar 2023 05:35:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233236AbjC0Jfi (ORCPT ); Mon, 27 Mar 2023 05:35:38 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A0449D1; Mon, 27 Mar 2023 02:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909736; x=1711445736; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=prR90xwsDCfDNxJjL9hXgdF5o6pMTiJmwWoTjfPFxLM=; b=CljRp6hOjNP5i0sF3nEbEXxKdHuJB1VLF/rvYH+WB8fK6p2hOHZBVabF ryTqtwWobx3xPjBgXJMBv/E9HTkPAHMHkfkncCO5CfBtnZBNFHCZU0V4P 4E3KsBApIIt945LWplFpMtIV1hkZUnMdB7ETfCHYGueSzdaHJo70jNUtt zo+ZFd73LfAF0+ErSFWN90pk3YnsoMH3lBTHcyaSdRTQGtSwCe7okn2MQ QPxU8hjVO0QLiAIROwapd6ZiOnRBzlNW6s7HF8tZDLrZmpvKyvYd5TLii 37Ola2Lh2wvSQUrcsHFUncwaakmljf4b5P0ZbhaBfeV5jWyS5J4POPraI Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879555" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879555" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554656" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554656" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:06 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 07/10] vfio: Accpet device file from vfio PCI hot reset path Date: Mon, 27 Mar 2023 02:34:55 -0700 Message-Id: <20230327093458.44939-8-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This extends both vfio_file_is_valid() and vfio_file_has_dev() to accept device file from the vfio PCI hot reset. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/vfio_main.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index fe7446805afd..ebbb6b91a498 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1154,13 +1154,23 @@ const struct file_operations vfio_device_fops = { .mmap = vfio_device_fops_mmap, }; +static struct vfio_device *vfio_device_from_file(struct file *file) +{ + struct vfio_device *device = file->private_data; + + if (file->f_op != &vfio_device_fops) + return NULL; + return device; +} + /** * vfio_file_is_valid - True if the file is valid vfio file * @file: VFIO group file or VFIO device file */ bool vfio_file_is_valid(struct file *file) { - return vfio_group_from_file(file); + return vfio_group_from_file(file) || + vfio_device_from_file(file); } EXPORT_SYMBOL_GPL(vfio_file_is_valid); @@ -1174,12 +1184,17 @@ EXPORT_SYMBOL_GPL(vfio_file_is_valid); bool vfio_file_has_dev(struct file *file, struct vfio_device *device) { struct vfio_group *group; + struct vfio_device *vdev; group = vfio_group_from_file(file); - if (!group) - return false; + if (group) + return vfio_group_has_dev(group, device); + + vdev = vfio_device_from_file(file); + if (vdev) + return vdev == device; - return vfio_group_has_dev(group, device); + return false; } EXPORT_SYMBOL_GPL(vfio_file_has_dev); From patchwork Mon Mar 27 09:34:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD530C7619A for ; Mon, 27 Mar 2023 09:36:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233422AbjC0JgA (ORCPT ); Mon, 27 Mar 2023 05:36:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233276AbjC0Jfk (ORCPT ); Mon, 27 Mar 2023 05:35:40 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D0DA46AA; Mon, 27 Mar 2023 02:35:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909737; x=1711445737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=j5nwiM6RBu75u+6frQmtDejnHzhEKYNU9/P5Y8zUpgE=; b=AtQ7bRMo1u1S5FLaNTpanYSchgcpup7BtqxLQOu2Aml67j3iehxOxMnQ AUTird1ErQio2+kFayd4uA9Bm5GZJR0dDZM7YZL+nM7D23c0k8JCNMy2S vs6FjHSjl/eM+HWjrGl9YvLErmiNo4qcr+3jggHgzsDJlNCvTmFOeOvMF lp5Kho83H8HIZ48K83FCWr9NJg/nWjaGaXRz1sKBXFieCaFDXlD2zWL1m e56B0l1rjZuPGCkVhoeI6R+pEP6V0BK2TLI2G9WMe2VhZ+mmdqeUcvNwI g6TUjHwtKwGzEWrxA6LL/3KslIAlnynQAwacuooxFb4HwG5vPjB7O9oPA A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879576" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879576" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554668" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554668" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:07 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 08/10] vfio/pci: Renaming for accepting device fd in hot reset path Date: Mon, 27 Mar 2023 02:34:56 -0700 Message-Id: <20230327093458.44939-9-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org No functional change is intended. Reviewed-by: Kevin Tian Reviewed-by: Jason Gunthorpe Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 52 ++++++++++++++++---------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2a510b71edcb..da6325008872 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -177,10 +177,10 @@ static void vfio_pci_probe_mmaps(struct vfio_pci_core_device *vdev) } } -struct vfio_pci_group_info; +struct vfio_pci_file_info; static void vfio_pci_dev_set_try_reset(struct vfio_device_set *dev_set); static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx); /* @@ -800,7 +800,7 @@ static int vfio_pci_fill_devs(struct pci_dev *pdev, void *data) return 0; } -struct vfio_pci_group_info { +struct vfio_pci_file_info { int count; struct file **files; }; @@ -1257,14 +1257,14 @@ static int vfio_pci_ioctl_get_pci_hot_reset_info( } static int -vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_hot_reset *hdr, - bool slot, - struct vfio_pci_hot_reset __user *arg) +vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset *hdr, + bool slot, + struct vfio_pci_hot_reset __user *arg) { - int32_t *group_fds; + int32_t *fds; struct file **files; - struct vfio_pci_group_info info; + struct vfio_pci_file_info info; int file_idx, count = 0, ret = 0; /* @@ -1281,17 +1281,17 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, if (hdr->count > count) return -EINVAL; - group_fds = kcalloc(hdr->count, sizeof(*group_fds), GFP_KERNEL); + fds = kcalloc(hdr->count, sizeof(*fds), GFP_KERNEL); files = kcalloc(hdr->count, sizeof(*files), GFP_KERNEL); - if (!group_fds || !files) { - kfree(group_fds); + if (!fds || !files) { + kfree(fds); kfree(files); return -ENOMEM; } - if (copy_from_user(group_fds, arg->group_fds, - hdr->count * sizeof(*group_fds))) { - kfree(group_fds); + if (copy_from_user(fds, arg->group_fds, + hdr->count * sizeof(*fds))) { + kfree(fds); kfree(files); return -EFAULT; } @@ -1301,7 +1301,7 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, * the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { - struct file *file = fget(group_fds[file_idx]); + struct file *file = fget(fds[file_idx]); if (!file) { ret = -EBADF; @@ -1318,9 +1318,9 @@ vfio_pci_ioctl_pci_hot_reset_groups(struct vfio_pci_core_device *vdev, files[file_idx] = file; } - kfree(group_fds); + kfree(fds); - /* release reference to groups on error */ + /* release reference to fds on error */ if (ret) goto hot_reset_release; @@ -1358,7 +1358,7 @@ static int vfio_pci_ioctl_pci_hot_reset(struct vfio_pci_core_device *vdev, return -ENODEV; if (hdr.count) - return vfio_pci_ioctl_pci_hot_reset_groups(vdev, &hdr, slot, arg); + return vfio_pci_ioctl_pci_hot_reset_files(vdev, &hdr, slot, arg); iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); @@ -2329,16 +2329,16 @@ const struct pci_error_handlers vfio_pci_core_err_handlers = { }; EXPORT_SYMBOL_GPL(vfio_pci_core_err_handlers); -static bool vfio_dev_in_groups(struct vfio_pci_core_device *vdev, - struct vfio_pci_group_info *groups) +static bool vfio_dev_in_files(struct vfio_pci_core_device *vdev, + struct vfio_pci_file_info *info) { unsigned int i; - if (!groups) + if (!info) return false; - for (i = 0; i < groups->count; i++) - if (vfio_file_has_dev(groups->files[i], &vdev->vdev)) + for (i = 0; i < info->count; i++) + if (vfio_file_has_dev(info->files[i], &vdev->vdev)) return true; return false; } @@ -2429,7 +2429,7 @@ static bool vfio_dev_in_iommufd_ctx(struct vfio_pci_core_device *vdev, * get each memory_lock. */ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, - struct vfio_pci_group_info *groups, + struct vfio_pci_file_info *info, struct iommufd_ctx *iommufd_ctx) { struct vfio_pci_core_device *cur_mem; @@ -2478,7 +2478,7 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * the calling device is in a singleton dev_set. */ if (cur_vma->vdev.open_count && - !vfio_dev_in_groups(cur_vma, groups) && + !vfio_dev_in_files(cur_vma, info) && !vfio_dev_in_iommufd_ctx(cur_vma, iommufd_ctx) && (dev_set->device_count > 1)) { ret = -EINVAL; From patchwork Mon Mar 27 09:34:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D753CC76195 for ; Mon, 27 Mar 2023 09:36:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233251AbjC0JgA (ORCPT ); Mon, 27 Mar 2023 05:36:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233045AbjC0Jfl (ORCPT ); Mon, 27 Mar 2023 05:35:41 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BB54526F; Mon, 27 Mar 2023 02:35:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909737; x=1711445737; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=v0D3R5+aGDWyOemoEsVYiHjHAhjBokxdDCY6e1VuJQQ=; b=BW/l3UFuhNbklpCD12ODObGkjlV8iPuxyHWaA5rtTmWWe92ZUBqbj5xJ Wk3/LDot0+80S0+LVtknbAoMOyRg4dlUi/AlgAe9fV06E67EeGkmA1dB6 hO50bSs++JfVkZzsxUELLo0m54YNKuuuDemH6/V/uPEzpYLBV4BK1PXfS H39G5r/PlP7+jDoZg9rQw9Dyzeqs1Cy/JSnl+s4xPMsSy1K8KPSfqSzid axkexpQGZD2ev1OHuMAUSL+ZB8ILHf1woFQnzDam7dHNsiumpuw07101z YuJSpVU0Lz/Yga2c6q7fsdL9SAx20kob9i3YoR8sEvePCuTsoFJkgEsCh g==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879588" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879588" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554680" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554680" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:08 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 09/10] vfio/pci: Accept device fd in VFIO_DEVICE_PCI_HOT_RESET ioctl Date: Mon, 27 Mar 2023 02:34:57 -0700 Message-Id: <20230327093458.44939-10-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now user can also provide an array of device fds as a 3rd method to verify the reset ownership. It's not useful at this point when the device fds are acquired via group fds. But it's necessary when moving to device cdev which allows the user to directly acquire device fds by skipping group. In that case this method can be used as a last resort when the preferred iommufd verification doesn't work, e.g. in noiommu usages. Clarify it in uAPI. Reviewed-by: Kevin Tian Signed-off-by: Yi Liu Reviewed-by: Jason Gunthorpe --- drivers/vfio/pci/vfio_pci_core.c | 9 +++++---- include/uapi/linux/vfio.h | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index da6325008872..19f5b075d70a 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1289,7 +1289,7 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, return -ENOMEM; } - if (copy_from_user(fds, arg->group_fds, + if (copy_from_user(fds, arg->fds, hdr->count * sizeof(*fds))) { kfree(fds); kfree(files); @@ -1297,8 +1297,8 @@ vfio_pci_ioctl_pci_hot_reset_files(struct vfio_pci_core_device *vdev, } /* - * Get the group file for each fd to ensure the group held across - * the reset + * Get the file for each fd to ensure the group/device file + * is held across the reset */ for (file_idx = 0; file_idx < hdr->count; file_idx++) { struct file *file = fget(fds[file_idx]); @@ -2469,7 +2469,8 @@ static int vfio_pci_dev_set_hot_reset(struct vfio_device_set *dev_set, * cannot race being opened by another user simultaneously. * * Otherwise all opened devices in the dev_set must be - * contained by the set of groups provided by the user. + * contained by the set of groups/devices provided by + * the user. * * If user provides a zero-length array, then all the * opened devices must be bound to a same iommufd_ctx. diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 17aa5d09db41..25432ef213ee 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -681,6 +681,7 @@ struct vfio_pci_hot_reset_info { * * The ownership can be proved by: * - An array of group fds + * - An array of device fds * - A zero-length array * * In the last case all affected devices which are opened by this user @@ -694,7 +695,7 @@ struct vfio_pci_hot_reset { __u32 argsz; __u32 flags; __u32 count; - __s32 group_fds[]; + __s32 fds[]; }; #define VFIO_DEVICE_PCI_HOT_RESET _IO(VFIO_TYPE, VFIO_BASE + 13) From patchwork Mon Mar 27 09:34:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 13188911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82C99C761AF for ; Mon, 27 Mar 2023 09:36:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233281AbjC0JgC (ORCPT ); Mon, 27 Mar 2023 05:36:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233319AbjC0Jfm (ORCPT ); Mon, 27 Mar 2023 05:35:42 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 682A519B2; Mon, 27 Mar 2023 02:35:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679909738; x=1711445738; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8gWbAJJlIuV/z8+WOO6T7u0CyD5t7kZGlKBcfpHfaUQ=; b=VHC7gzRUIcMpSquCftVNP3Lv/GY3xt92mIgfbflStgfLceTHrSSDRHAO myrgnsrP0eo5Gj2Y+nY/oeTjChyR3AZwcbWfgxDPItIxYF79JbH4d0saF 2zRON3H/zVjll4pSBsdTEl6PXyooIlRvFCtFfwRU9hi1uOYBWNgGhvyBg AmrM56E6oj83vCeMkKppXPmg87Wu6Jb2ylkEbYkgMl7a7BND71vYOuycD nSXg6vJZmUwccL6imqqJJOZGkDc3ms27U/zCVNvyR+zW3OG9Uf9vbQRPe Pu8JlSLH1uH4crGYur1YpG2aGWDugYKu3eAOOtXXlpHycp5jYmPfaCybl A==; X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="319879604" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="319879604" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Mar 2023 02:35:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10661"; a="633554687" X-IronPort-AV: E=Sophos;i="5.98,294,1673942400"; d="scan'208";a="633554687" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga003.jf.intel.com with ESMTP; 27 Mar 2023 02:35:10 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com Cc: joro@8bytes.org, robin.murphy@arm.com, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, linux-s390@vger.kernel.org, xudong.hao@intel.com, yan.y.zhao@intel.com, terrence.xu@intel.com, yanting.jiang@intel.com Subject: [PATCH v2 10/10] vfio/pci: Add VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO Date: Mon, 27 Mar 2023 02:34:58 -0700 Message-Id: <20230327093458.44939-11-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230327093458.44939-1-yi.l.liu@intel.com> References: <20230327093458.44939-1-yi.l.liu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is a preparation for vfio device cdev as cdev gives userspace the capability to open device cdev fd and management stack (e.g. libvirt) could pass the device fd to the actual user (e.g. QEMU). As a result, the actual user has no idea about the device's bus:devfn information. This is a problem when user uses VFIO_DEVICE_GET_PCI_HOT_RESET_INFO to know the hot reset affected device scope as this ioctl returns bus:devfn info. For the fd passing usage, the acutal user cannot map the bus:devfn to the devices it has opened via the fd passed from management stack. So a new ioctl is required. This new ioctl reports the list of iommufd dev_id that is opened by the user. If there is affected device that is not bound to vfio driver or opened by another user, this command shall fail with -EPERM. For the noiommu mode in the vfio device cdev path, this shall fail as no dev_id would be generated, hence nothing to report. This ioctl is useless to the users that open vfio group as such users have no idea about the iommufd dev_id and it can use the existing VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. The user that uses the traditional mode vfio group/container would be failed if invoking this ioctl. But the user that uses the iommufd compat mode vfio group/container shall succeed. This is harmless as long as user cannot make use of it and should use VFIO_DEVICE_GET_PCI_HOT_RESET_INFO. Signed-off-by: Yi Liu Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_core.c | 98 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 47 +++++++++++++++ 2 files changed, 145 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 19f5b075d70a..45edf4e9b98b 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1181,6 +1181,102 @@ static int vfio_pci_ioctl_reset(struct vfio_pci_core_device *vdev, return ret; } +static struct pci_dev * +vfio_pci_dev_set_resettable(struct vfio_device_set *dev_set); + +static int vfio_pci_ioctl_get_pci_hot_reset_group_info( + struct vfio_pci_core_device *vdev, + struct vfio_pci_hot_reset_group_info __user *arg) +{ + unsigned long minsz = + offsetofend(struct vfio_pci_hot_reset_group_info, count); + struct vfio_pci_hot_reset_group_info hdr; + struct iommufd_ctx *iommufd, *cur_iommufd; + u32 count = 0, index = 0, *devices = NULL; + struct vfio_pci_core_device *cur; + bool slot = false; + int ret = 0; + + if (copy_from_user(&hdr, arg, minsz)) + return -EFAULT; + + if (hdr.argsz < minsz) + return -EINVAL; + + hdr.flags = 0; + + /* Can we do a slot or bus reset or neither? */ + if (!pci_probe_reset_slot(vdev->pdev->slot)) + slot = true; + else if (pci_probe_reset_bus(vdev->pdev->bus)) + return -ENODEV; + + mutex_lock(&vdev->vdev.dev_set->lock); + if (!vfio_pci_dev_set_resettable(vdev->vdev.dev_set)) { + ret = -EPERM; + goto out_unlock; + } + + iommufd = vfio_iommufd_physical_ictx(&vdev->vdev); + if (!iommufd) { + ret = -EPERM; + goto out_unlock; + } + + /* How many devices are affected? */ + ret = vfio_pci_for_each_slot_or_bus(vdev->pdev, vfio_pci_count_devs, + &count, slot); + if (ret) + goto out_unlock; + + WARN_ON(!count); /* Should always be at least one */ + + /* + * If there's enough space, fill it now, otherwise return -ENOSPC and + * the number of devices affected. + */ + if (hdr.argsz < sizeof(hdr) + (count * sizeof(*devices))) { + ret = -ENOSPC; + hdr.count = count; + goto reset_info_exit; + } + + devices = kcalloc(count, sizeof(*devices), GFP_KERNEL); + if (!devices) { + ret = -ENOMEM; + goto reset_info_exit; + } + + list_for_each_entry(cur, &vdev->vdev.dev_set->device_list, vdev.dev_set_list) { + cur_iommufd = vfio_iommufd_physical_ictx(&cur->vdev); + if (cur->vdev.open_count) { + if (cur_iommufd != iommufd) { + ret = -EPERM; + break; + } + ret = vfio_iommufd_physical_devid(&cur->vdev, &devices[index]); + if (ret) + break; + index++; + } + } + +reset_info_exit: + if (copy_to_user(arg, &hdr, minsz)) + ret = -EFAULT; + + if (!ret) { + if (copy_to_user(&arg->devices, devices, + hdr.count * sizeof(*devices))) + ret = -EFAULT; + } + + kfree(devices); +out_unlock: + mutex_unlock(&vdev->vdev.dev_set->lock); + return ret; +} + static int vfio_pci_ioctl_get_pci_hot_reset_info( struct vfio_pci_core_device *vdev, struct vfio_pci_hot_reset_info __user *arg) @@ -1404,6 +1500,8 @@ long vfio_pci_core_ioctl(struct vfio_device *core_vdev, unsigned int cmd, return vfio_pci_ioctl_get_irq_info(vdev, uarg); case VFIO_DEVICE_GET_PCI_HOT_RESET_INFO: return vfio_pci_ioctl_get_pci_hot_reset_info(vdev, uarg); + case VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO: + return vfio_pci_ioctl_get_pci_hot_reset_group_info(vdev, uarg); case VFIO_DEVICE_GET_REGION_INFO: return vfio_pci_ioctl_get_region_info(vdev, uarg); case VFIO_DEVICE_IOEVENTFD: diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 25432ef213ee..61b801dfd40b 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -669,6 +669,53 @@ struct vfio_pci_hot_reset_info { #define VFIO_DEVICE_GET_PCI_HOT_RESET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12) +/** + * VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12, + * struct vfio_pci_hot_reset_group_info) + * + * This is used in the vfio device cdev mode. It returns the list of + * affected devices (represented by iommufd dev_id) when hot reset is + * issued on the current device with which this ioctl is invoked. It + * only includes the devices that are opened by the current user in the + * time of this command is invoked. This list may change when user opens + * new device or close opened device, hence user should re-invoke it + * after open/close devices. This command has no guarantee on the result + * of VFIO_DEVICE_PCI_HOT_RESET since the not-opened affected device can + * be by other users in the window between the two ioctls. If the affected + * devices are opened by multiple users, the VFIO_DEVICE_PCI_HOT_RESET + * shall fail, detail can check the description of VFIO_DEVICE_PCI_HOT_RESET. + * + * For the users that open vfio group/container, this ioctl is useless as + * they have no idea about the iommufd dev_id returned by this ioctl. For + * the users of the traditional mode vfio group/container, this ioctl will + * fail as this mode does not use iommufd hence no dev_id to report back. + * For the users of the iommufd compat mode vfio group/container, this ioctl + * would succeed as this mode uses iommufd as container fd. But such users + * still have no idea about the iommufd dev_id as the dev_id is only stored + * in kernel in this mode. For the users of the vfio group/container, the + * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO should be used to know the hot reset + * affected devices. + * + * Return: 0 on success, -errno on failure: + * -enospc = insufficient buffer; + * -enodev = unsupported for device; + * -eperm = no permission for device, this error comes: + * - when there are affected devices that are opened but + * not bound to the same iommufd with the current device + * with which this ioctl is invoked, + * - there are affected devices that are not bound to vfio + * driver yet. + * - no valid iommufd is bound (e.g. noiommu mode) + */ +struct vfio_pci_hot_reset_group_info { + __u32 argsz; + __u32 flags; + __u32 count; + __u32 devices[]; +}; + +#define VFIO_DEVICE_GET_PCI_HOT_RESET_GROUP_INFO _IO(VFIO_TYPE, VFIO_BASE + 18) + /** * VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13, * struct vfio_pci_hot_reset)