From patchwork Tue Feb 11 11:45:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Ryabinin X-Patchwork-Id: 13970325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BBA76C0219B for ; Tue, 11 Feb 2025 16:57:35 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CF8E410E714; Tue, 11 Feb 2025 16:57:33 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.com header.i=@yandex-team.com header.b="LJ5WKFHm"; dkim-atps=neutral Received: from forwardcorp1a.mail.yandex.net (forwardcorp1a.mail.yandex.net [178.154.239.72]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8184210E2E8; Tue, 11 Feb 2025 11:53:11 +0000 (UTC) Received: from mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net [IPv6:2a02:6b8:c0f:1286:0:640:6f2b:0]) by forwardcorp1a.mail.yandex.net (Yandex) with ESMTPS id A65BE614BD; Tue, 11 Feb 2025 14:46:00 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [10.214.35.248]) by mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id njMaW61IYa60-0cIaOumc; Tue, 11 Feb 2025 14:45:59 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1739274360; bh=y1+8TKaZyt8/6ij0O/nQoH47E1MV/9ZBI4/TT9VKt0Q=; h=Message-ID:Date:Cc:Subject:To:From; b=LJ5WKFHmyCrCn1KYL3qcylN81InuVfSerkMsn+VBoeKh8/wdxrbUdzZdQ+PpMKMyh oGWt9zOHcNbRA3SiMQ2rM6FIaaD8FM4h8AwmMdD3qpAfcSUkrKnO1pBCwXvTw8TKGl FAEAxOhG/j/IwM2mkQ+KPz4ubeN23snm5HpH+GaQ= Authentication-Results: mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: Alex Williamson Cc: Zhenyu Wang , Zhi Wang , Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , Tvrtko Ursulin , David Airlie , Simona Vetter , Matthew Rosato , Jason Gunthorpe , Kevin Tian , Yi Liu , intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Andrey Ryabinin , stable@vger.kernel.org Subject: [PATCH 1/2] drm/i915/gvt: Store ->kvm reference in intel_vgpu struct. Date: Tue, 11 Feb 2025 12:45:43 +0100 Message-ID: <20250211114544.17845-1-arbn@yandex-team.com> X-Mailer: git-send-email 2.45.3 MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 11 Feb 2025 16:57:31 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" 'vfio_device' keeps the ->kvm pointer with elevated counter from the first open of the device up until the last close(). So the kvm struct and its dependencies (kvm kthreads, cgroups ...) kept alive even for VFIO device that don't need ->kvm. Copy ->kvm pointer from the vfio_device struct and store it in the 'intel_vgpu'. Note that kvm_page_track_[un]register_notifier() already does get/put calls, keeping the kvm struct alive. This will allow to release ->kvm from the vfio_device righ after the first open call, so that devices not using kvm not keeping it alive. Devices that are using kvm (like intel_vgpu) will be expected to mange the lifetime of the kvm struct by themselves. Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock") Cc: Signed-off-by: Andrey Ryabinin --- drivers/gpu/drm/i915/gvt/gvt.h | 1 + drivers/gpu/drm/i915/gvt/kvmgt.c | 14 +++++++------- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/i915/gvt/gvt.h b/drivers/gpu/drm/i915/gvt/gvt.h index 2c95aeef4e41..6c62467df22c 100644 --- a/drivers/gpu/drm/i915/gvt/gvt.h +++ b/drivers/gpu/drm/i915/gvt/gvt.h @@ -232,6 +232,7 @@ struct intel_vgpu { unsigned long nr_cache_entries; struct mutex cache_lock; + struct kvm *kvm; struct kvm_page_track_notifier_node track_node; #define NR_BKT (1 << 18) struct hlist_head ptable[NR_BKT]; diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c index b27ff77bfb50..cf418e2c560d 100644 --- a/drivers/gpu/drm/i915/gvt/kvmgt.c +++ b/drivers/gpu/drm/i915/gvt/kvmgt.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -649,7 +650,7 @@ static bool __kvmgt_vgpu_exist(struct intel_vgpu *vgpu) if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, itr->status)) continue; - if (vgpu->vfio_device.kvm == itr->vfio_device.kvm) { + if (vgpu->kvm == itr->kvm) { ret = true; goto out; } @@ -664,13 +665,13 @@ static int intel_vgpu_open_device(struct vfio_device *vfio_dev) struct intel_vgpu *vgpu = vfio_dev_to_vgpu(vfio_dev); int ret; + vgpu->kvm = vgpu->vfio_device.kvm; if (__kvmgt_vgpu_exist(vgpu)) return -EEXIST; vgpu->track_node.track_write = kvmgt_page_track_write; vgpu->track_node.track_remove_region = kvmgt_page_track_remove_region; - ret = kvm_page_track_register_notifier(vgpu->vfio_device.kvm, - &vgpu->track_node); + ret = kvm_page_track_register_notifier(vgpu->kvm, &vgpu->track_node); if (ret) { gvt_vgpu_err("KVM is required to use Intel vGPU\n"); return ret; @@ -707,8 +708,7 @@ static void intel_vgpu_close_device(struct vfio_device *vfio_dev) debugfs_lookup_and_remove(KVMGT_DEBUGFS_FILENAME, vgpu->debugfs); - kvm_page_track_unregister_notifier(vgpu->vfio_device.kvm, - &vgpu->track_node); + kvm_page_track_unregister_notifier(vgpu->kvm, &vgpu->track_node); kvmgt_protect_table_destroy(vgpu); gvt_cache_destroy(vgpu); @@ -1560,7 +1560,7 @@ int intel_gvt_page_track_add(struct intel_vgpu *info, u64 gfn) if (kvmgt_gfn_is_write_protected(info, gfn)) return 0; - r = kvm_write_track_add_gfn(info->vfio_device.kvm, gfn); + r = kvm_write_track_add_gfn(info->kvm, gfn); if (r) return r; @@ -1578,7 +1578,7 @@ int intel_gvt_page_track_remove(struct intel_vgpu *info, u64 gfn) if (!kvmgt_gfn_is_write_protected(info, gfn)) return 0; - r = kvm_write_track_remove_gfn(info->vfio_device.kvm, gfn); + r = kvm_write_track_remove_gfn(info->kvm, gfn); if (r) return r; From patchwork Tue Feb 11 11:45:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Ryabinin X-Patchwork-Id: 13970324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83796C021A1 for ; Tue, 11 Feb 2025 16:57:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C747410E70E; Tue, 11 Feb 2025 16:57:31 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (1024-bit key; unprotected) header.d=yandex-team.com header.i=@yandex-team.com header.b="doARputU"; dkim-atps=neutral X-Greylist: delayed 427 seconds by postgrey-1.36 at gabe; Tue, 11 Feb 2025 11:53:11 UTC Received: from forwardcorp1a.mail.yandex.net (forwardcorp1a.mail.yandex.net [178.154.239.72]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8185D10E69A; Tue, 11 Feb 2025 11:53:11 +0000 (UTC) Received: from mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net [IPv6:2a02:6b8:c0f:1286:0:640:6f2b:0]) by forwardcorp1a.mail.yandex.net (Yandex) with ESMTPS id BB47F60CBC; Tue, 11 Feb 2025 14:46:02 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [10.214.35.248]) by mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id njMaW61IYa60-f300HgHw; Tue, 11 Feb 2025 14:46:02 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1739274362; bh=N/f4M/1YdrDaXLy1gcP2Kg22i5sVeuoh7Qx0Xv+wu0U=; h=Message-ID:Date:In-Reply-To:Cc:Subject:References:To:From; b=doARputUw27HsyLKkbE04O5ibJYLITuopDPsmuZ9aczHNvBTKndE0yZghx202ewCN R0AlGAPmGGvsezMwhUJQ/uepQErBEbYbxzVqcsT+vv5AwhPpqZC4vKmvEML7VbEde/ eXAmQ0zBtCojSSOabGstArsm3ER7ferxhDM5F2Bk= Authentication-Results: mail-nwsmtp-smtp-corp-main-83.vla.yp-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: Alex Williamson Cc: Zhenyu Wang , Zhi Wang , Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , Tvrtko Ursulin , David Airlie , Simona Vetter , Matthew Rosato , Jason Gunthorpe , Kevin Tian , Yi Liu , intel-gvt-dev@lists.freedesktop.org, intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Andrey Ryabinin , stable@vger.kernel.org Subject: [PATCH 2/2] vfio: Release KVM pointer after the first device open. Date: Tue, 11 Feb 2025 12:45:44 +0100 Message-ID: <20250211114544.17845-2-arbn@yandex-team.com> X-Mailer: git-send-email 2.45.3 In-Reply-To: <20250211114544.17845-1-arbn@yandex-team.com> References: <20250211114544.17845-1-arbn@yandex-team.com> MIME-Version: 1.0 X-Mailman-Approved-At: Tue, 11 Feb 2025 16:57:31 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Commit 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock") made vfio_device to hold KVM struct up until device's close() call. This lead to a unrleased KVM struct which holds KVM kthreads and related cgroups after VM with VFIO device migrates to from one KVM instance to another on the same host. Since all drivers, that require 'kvm' (vfio-ap/intel_vgp/vfio-pci zdev) already handle 'kvm' pointer by themselves we can just drop 'kvm' reference right after first vfio_df_open() call. This will release 'kvm' struct and dependent resources for drivers that don't require it after KVM detached from a device (KVM_DEV_VFIO_FILE_DEL). Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock") Cc: Signed-off-by: Andrey Ryabinin --- drivers/vfio/device_cdev.c | 11 ++++++----- drivers/vfio/group.c | 31 ++++++++++++++----------------- 2 files changed, 20 insertions(+), 22 deletions(-) diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c index bb1817bd4ff3..339b69c43300 100644 --- a/drivers/vfio/device_cdev.c +++ b/drivers/vfio/device_cdev.c @@ -103,14 +103,16 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, /* * Before the device open, get the KVM pointer currently * associated with the device file (if there is) and obtain - * a reference. This reference is held until device closed. + * a reference and release it right after vfio_df_open() bellow. + * The device driver wishes to use KVM must obtain a reference and + * release it on close. * Save the pointer in the device for use by drivers. */ vfio_df_get_kvm_safe(df); - ret = vfio_df_open(df); + vfio_device_put_kvm(device); if (ret) - goto out_put_kvm; + goto out_put_iommufd; ret = copy_to_user(&arg->out_devid, &df->devid, sizeof(df->devid)) ? -EFAULT : 0; @@ -128,8 +130,7 @@ long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df, out_close_device: vfio_df_close(df); -out_put_kvm: - vfio_device_put_kvm(device); +out_put_iommufd: iommufd_ctx_put(df->iommufd); df->iommufd = NULL; out_unlock: diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c index 49559605177e..872cfd795f99 100644 --- a/drivers/vfio/group.c +++ b/drivers/vfio/group.c @@ -175,15 +175,6 @@ static int vfio_df_group_open(struct vfio_device_file *df) mutex_lock(&device->dev_set->lock); - /* - * Before the first device open, get the KVM pointer currently - * associated with the group (if there is one) and obtain a reference - * now that will be held until the open_count reaches 0 again. Save - * the pointer in the device for use by drivers. - */ - if (device->open_count == 0) - vfio_device_group_get_kvm_safe(device); - df->iommufd = device->group->iommufd; if (df->iommufd && vfio_device_is_noiommu(device) && device->open_count == 0) { /* @@ -196,12 +187,23 @@ static int vfio_df_group_open(struct vfio_device_file *df) ret = -EPERM; else ret = 0; - goto out_put_kvm; + goto out_iommufd; } + /* + * Before the first device open, get the KVM pointer currently + * associated with the group (if there is one) and obtain a reference + * now that will be released right after vfio_df_open() bellow. + * The device driver wishes to use KVM must obtain a reference and + * release it on close. + */ + if (device->open_count == 0) + vfio_device_group_get_kvm_safe(device); + ret = vfio_df_open(df); + vfio_device_put_kvm(device); if (ret) - goto out_put_kvm; + goto out_iommufd; if (df->iommufd && device->open_count == 1) { ret = vfio_iommufd_compat_attach_ioas(device, df->iommufd); @@ -221,10 +223,8 @@ static int vfio_df_group_open(struct vfio_device_file *df) out_close_device: vfio_df_close(df); -out_put_kvm: +out_iommufd: df->iommufd = NULL; - if (device->open_count == 0) - vfio_device_put_kvm(device); mutex_unlock(&device->dev_set->lock); out_unlock: mutex_unlock(&device->group->group_lock); @@ -241,9 +241,6 @@ void vfio_df_group_close(struct vfio_device_file *df) vfio_df_close(df); df->iommufd = NULL; - if (device->open_count == 0) - vfio_device_put_kvm(device); - mutex_unlock(&device->dev_set->lock); mutex_unlock(&device->group->group_lock); }