From patchwork Tue Jan 7 17:32:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1116E77197 for ; Tue, 7 Jan 2025 14:22:32 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5646A10E6EB; Tue, 7 Jan 2025 14:22:32 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="cG3rwd0j"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id DF06610E6EB for ; Tue, 7 Jan 2025 14:22:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259750; x=1767795750; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hq+g4oc3QklxrpvnQp7R4QV+2/xicfwVUXoXC54YJ74=; b=cG3rwd0jeelOsaa+9lpFXauHxVCmcdcgWdKfqwWXOsOhTWrp66VBeXGs daPr6Zq6+Lf720Ga1RSQeUuN/HcdOUaSKV0x9bI8WzJfSwdQwjRkeOACs omzNCWzkwyBHlU6veCuA4FuNukpgz5ZfaagyLuZ3bS3c+xSdjrikK0t/N 3/+IyPdlIvbAFQm1Z02qUGezN2E6su9jmUs/0b+hX4X1MvTWHFDjBj9Jf WIiY+xJg37mGchTlPWi6E3xs7HoIpNpw5mVZzuDamQMuUVdSPcNHhzKIB MXRxUnjCxC0h/bGaSdiFh58S3fnOSaXltF51sGR8SoE5XI4t5yYQ/mDuS g==; X-CSE-ConnectionGUID: rggryS7sSJOhzk1eBzFC0w== X-CSE-MsgGUID: cD8aBe6ZQPqswhU692JLew== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324419" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324419" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:30 -0800 X-CSE-ConnectionGUID: 3ODmwknsTE6s4jgcRphpcw== X-CSE-MsgGUID: kuke3rV0Swy18xvJYsvQCA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635396" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:29 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 01/14] accel/ivpu: Separate DB ID and CMDQ ID allocations from CMDQ allocation Date: Tue, 7 Jan 2025 18:32:24 +0100 Message-ID: <20250107173238.381120-2-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Move doorbell ID and command queue ID XArray allocations from command queue memory allocation function. This will allows IDs allocations to be done without the need for actual memory allocation. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_job.c | 88 +++++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 24 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index 7149312f16e1..98e53cb38ecd 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -83,23 +83,9 @@ static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) if (!cmdq) return NULL; - ret = xa_alloc_cyclic(&vdev->db_xa, &cmdq->db_id, NULL, vdev->db_limit, &vdev->db_next, - GFP_KERNEL); - if (ret < 0) { - ivpu_err(vdev, "Failed to allocate doorbell id: %d\n", ret); - goto err_free_cmdq; - } - - ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &cmdq->id, cmdq, file_priv->cmdq_limit, - &file_priv->cmdq_id_next, GFP_KERNEL); - if (ret < 0) { - ivpu_err(vdev, "Failed to allocate command queue id: %d\n", ret); - goto err_erase_db_xa; - } - cmdq->mem = ivpu_bo_create_global(vdev, SZ_4K, DRM_IVPU_BO_WC | DRM_IVPU_BO_MAPPABLE); if (!cmdq->mem) - goto err_erase_cmdq_xa; + goto err_free_cmdq; ret = ivpu_preemption_buffers_create(vdev, file_priv, cmdq); if (ret) @@ -107,10 +93,6 @@ static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) return cmdq; -err_erase_cmdq_xa: - xa_erase(&file_priv->cmdq_xa, cmdq->id); -err_erase_db_xa: - xa_erase(&vdev->db_xa, cmdq->db_id); err_free_cmdq: kfree(cmdq); return NULL; @@ -233,30 +215,88 @@ static int ivpu_cmdq_fini(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cm return 0; } +static int ivpu_db_id_alloc(struct ivpu_device *vdev, u32 *db_id) +{ + int ret; + u32 id; + + ret = xa_alloc_cyclic(&vdev->db_xa, &id, NULL, vdev->db_limit, &vdev->db_next, GFP_KERNEL); + if (ret < 0) + return ret; + + *db_id = id; + return 0; +} + +static int ivpu_cmdq_id_alloc(struct ivpu_file_priv *file_priv, u32 *cmdq_id) +{ + int ret; + u32 id; + + ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &id, NULL, file_priv->cmdq_limit, + &file_priv->cmdq_id_next, GFP_KERNEL); + if (ret < 0) + return ret; + + *cmdq_id = id; + return 0; +} + static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u8 priority) { + struct ivpu_device *vdev = file_priv->vdev; struct ivpu_cmdq *cmdq; - unsigned long cmdq_id; + unsigned long id; int ret; lockdep_assert_held(&file_priv->lock); - xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) + xa_for_each(&file_priv->cmdq_xa, id, cmdq) if (cmdq->priority == priority) break; if (!cmdq) { cmdq = ivpu_cmdq_alloc(file_priv); - if (!cmdq) + if (!cmdq) { + ivpu_err(vdev, "Failed to allocate command queue\n"); return NULL; + } + + ret = ivpu_db_id_alloc(vdev, &cmdq->db_id); + if (ret) { + ivpu_err(file_priv->vdev, "Failed to allocate doorbell ID: %d\n", ret); + goto err_free_cmdq; + } + + ret = ivpu_cmdq_id_alloc(file_priv, &cmdq->id); + if (ret) { + ivpu_err(vdev, "Failed to allocate command queue ID: %d\n", ret); + goto err_erase_db_id; + } + cmdq->priority = priority; + ret = xa_err(xa_store(&file_priv->cmdq_xa, cmdq->id, cmdq, GFP_KERNEL)); + if (ret) { + ivpu_err(vdev, "Failed to store command queue in cmdq_xa: %d\n", ret); + goto err_erase_cmdq_id; + } } ret = ivpu_cmdq_init(file_priv, cmdq, priority); - if (ret) - return NULL; + if (ret) { + ivpu_err(vdev, "Failed to initialize command queue: %d\n", ret); + goto err_free_cmdq; + } return cmdq; + +err_erase_cmdq_id: + xa_erase(&file_priv->cmdq_xa, cmdq->id); +err_erase_db_id: + xa_erase(&vdev->db_xa, cmdq->db_id); +err_free_cmdq: + ivpu_cmdq_free(file_priv, cmdq); + return NULL; } void ivpu_cmdq_release_all_locked(struct ivpu_file_priv *file_priv) From patchwork Tue Jan 7 17:32:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92A1AE77199 for ; Tue, 7 Jan 2025 14:22:34 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EA85110E6F1; Tue, 7 Jan 2025 14:22:33 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="LZvELwux"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id ED95110E6F1 for ; Tue, 7 Jan 2025 14:22:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259753; x=1767795753; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P0yB0j2VoTKpHq5isI5auP3Xy9H5auQKgT29vvJeMa4=; b=LZvELwuxRk2D20Oucn4AD2khOoKbaA83QU6wnFNJtqmNV2NiffxHRn5P 9LjJfQkol13xLQg4cznh3HQiozBZC3IFcx4UyjhDAAFnCzaDyz9JWLJIS 72ajlFtzxhtgiDn1PNbC7PST1qtU45JXYqZUV1v/sDzayGWLbjZ6cF8nr ximNJfxpYYm8kG3wVqc5SU0Ff90MT/sZJDacoNvrKvSR1UuWlm11jBACF 4o/JabOWlv9Ebdw3/7dfqN2C0/lL43YVWUA7o2yt/QFUJTG9EjE3gsx2N Qk+Qx/ODTO0bsCeg5gR8/Qarwy1pRqAwY8RBGiB6/6LcTQDN+5Scw60k+ g==; X-CSE-ConnectionGUID: 7+k9J/YbTGuo99SvS8XS3Q== X-CSE-MsgGUID: bIgRx5IOTDGiBs3N+IkqSw== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324425" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324425" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:32 -0800 X-CSE-ConnectionGUID: sG7xWJnJTXmsn3GsSkvMCg== X-CSE-MsgGUID: thG6Ng3YSPywet8KPsOgoQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635410" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:31 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 02/14] accel/ivpu: Add API for command queue create/destroy/submit Date: Tue, 7 Jan 2025 18:32:25 +0100 Message-ID: <20250107173238.381120-3-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Implement support for explicit command queue management. To allow more flexible control over command queues add capabilities to create, destroy and submit jobs to specific command queues. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_drv.c | 6 + drivers/accel/ivpu/ivpu_job.c | 369 +++++++++++++++++++++------------- drivers/accel/ivpu/ivpu_job.h | 5 +- include/uapi/drm/ivpu_accel.h | 84 ++++++++ 4 files changed, 319 insertions(+), 145 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_drv.c b/drivers/accel/ivpu/ivpu_drv.c index 1e8ffbe25eee..f4171536640b 100644 --- a/drivers/accel/ivpu/ivpu_drv.c +++ b/drivers/accel/ivpu/ivpu_drv.c @@ -137,6 +137,9 @@ static int ivpu_get_capabilities(struct ivpu_device *vdev, struct drm_ivpu_param case DRM_IVPU_CAP_DMA_MEMORY_RANGE: args->value = 1; break; + case DRM_IVPU_CAP_MANAGE_CMDQ: + args->value = 1; + break; default: return -EINVAL; } @@ -310,6 +313,9 @@ static const struct drm_ioctl_desc ivpu_drm_ioctls[] = { DRM_IOCTL_DEF_DRV(IVPU_METRIC_STREAMER_GET_DATA, ivpu_ms_get_data_ioctl, 0), DRM_IOCTL_DEF_DRV(IVPU_METRIC_STREAMER_STOP, ivpu_ms_stop_ioctl, 0), DRM_IOCTL_DEF_DRV(IVPU_METRIC_STREAMER_GET_INFO, ivpu_ms_get_info_ioctl, 0), + DRM_IOCTL_DEF_DRV(IVPU_CMDQ_CREATE, ivpu_cmdq_create_ioctl, 0), + DRM_IOCTL_DEF_DRV(IVPU_CMDQ_DESTROY, ivpu_cmdq_destroy_ioctl, 0), + DRM_IOCTL_DEF_DRV(IVPU_CMDQ_SUBMIT, ivpu_cmdq_submit_ioctl, 0), }; static int ivpu_wait_for_ready(struct ivpu_device *vdev) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index 98e53cb38ecd..43507d3aea51 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -100,15 +100,43 @@ static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) static void ivpu_cmdq_free(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) { - if (!cmdq) - return; - ivpu_preemption_buffers_free(file_priv->vdev, file_priv, cmdq); ivpu_bo_free(cmdq->mem); - xa_erase(&file_priv->vdev->db_xa, cmdq->db_id); kfree(cmdq); } +static struct ivpu_cmdq *ivpu_cmdq_create(struct ivpu_file_priv *file_priv, u8 priority, + bool is_legacy) +{ + struct ivpu_device *vdev = file_priv->vdev; + struct ivpu_cmdq *cmdq = NULL; + int ret; + + lockdep_assert_held(&file_priv->lock); + + cmdq = ivpu_cmdq_alloc(file_priv); + if (!cmdq) { + ivpu_err(vdev, "Failed to allocate command queue\n"); + return NULL; + } + + cmdq->priority = priority; + cmdq->is_legacy = is_legacy; + + ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &cmdq->id, cmdq, file_priv->cmdq_limit, + &file_priv->cmdq_id_next, GFP_KERNEL); + if (ret < 0) { + ivpu_err(vdev, "Failed to allocate command queue ID: %d\n", ret); + goto err_free_cmdq; + } + + return cmdq; + +err_free_cmdq: + ivpu_cmdq_free(file_priv, cmdq); + return NULL; +} + static int ivpu_hws_cmdq_init(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq, u16 engine, u8 priority) { @@ -134,6 +162,13 @@ static int ivpu_register_db(struct ivpu_file_priv *file_priv, struct ivpu_cmdq * struct ivpu_device *vdev = file_priv->vdev; int ret; + ret = xa_alloc_cyclic(&vdev->db_xa, &cmdq->db_id, NULL, vdev->db_limit, &vdev->db_next, + GFP_KERNEL); + if (ret < 0) { + ivpu_err(vdev, "Failed to allocate doorbell ID: %d\n", ret); + return ret; + } + if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) ret = ivpu_jsm_hws_register_db(vdev, file_priv->ctx.id, cmdq->id, cmdq->db_id, cmdq->mem->vpu_addr, ivpu_bo_size(cmdq->mem)); @@ -142,41 +177,52 @@ static int ivpu_register_db(struct ivpu_file_priv *file_priv, struct ivpu_cmdq * cmdq->mem->vpu_addr, ivpu_bo_size(cmdq->mem)); if (!ret) - ivpu_dbg(vdev, JOB, "DB %d registered to cmdq %d ctx %d\n", - cmdq->db_id, cmdq->id, file_priv->ctx.id); + ivpu_dbg(vdev, JOB, "DB %d registered to cmdq %d ctx %d priority %d\n", + cmdq->db_id, cmdq->id, file_priv->ctx.id, cmdq->priority); + else + xa_erase(&vdev->db_xa, cmdq->db_id); return ret; } -static int -ivpu_cmdq_init(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq, u8 priority) +static void ivpu_cmdq_jobq_init(struct ivpu_device *vdev, struct vpu_job_queue *jobq) +{ + jobq->header.engine_idx = VPU_ENGINE_COMPUTE; + jobq->header.head = 0; + jobq->header.tail = 0; + + if (ivpu_test_mode & IVPU_TEST_MODE_TURBO) { + ivpu_dbg(vdev, JOB, "Turbo mode enabled"); + jobq->header.flags = VPU_JOB_QUEUE_FLAGS_TURBO_MODE; + } + + wmb(); /* Flush WC buffer for jobq->header */ +} + +static inline u32 ivpu_cmdq_get_entry_count(struct ivpu_cmdq *cmdq) +{ + size_t size = ivpu_bo_size(cmdq->mem) - sizeof(struct vpu_job_queue_header); + + return size / sizeof(struct vpu_job_queue_entry); +} + +static int ivpu_cmdq_register(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) { struct ivpu_device *vdev = file_priv->vdev; - struct vpu_job_queue_header *jobq_header; int ret; lockdep_assert_held(&file_priv->lock); - if (cmdq->db_registered) + if (cmdq->db_id) return 0; - cmdq->entry_count = (u32)((ivpu_bo_size(cmdq->mem) - sizeof(struct vpu_job_queue_header)) / - sizeof(struct vpu_job_queue_entry)); - + cmdq->entry_count = ivpu_cmdq_get_entry_count(cmdq); cmdq->jobq = (struct vpu_job_queue *)ivpu_bo_vaddr(cmdq->mem); - jobq_header = &cmdq->jobq->header; - jobq_header->engine_idx = VPU_ENGINE_COMPUTE; - jobq_header->head = 0; - jobq_header->tail = 0; - if (ivpu_test_mode & IVPU_TEST_MODE_TURBO) { - ivpu_dbg(vdev, JOB, "Turbo mode enabled"); - jobq_header->flags = VPU_JOB_QUEUE_FLAGS_TURBO_MODE; - } - wmb(); /* Flush WC buffer for jobq->header */ + ivpu_cmdq_jobq_init(vdev, cmdq->jobq); if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) { - ret = ivpu_hws_cmdq_init(file_priv, cmdq, VPU_ENGINE_COMPUTE, priority); + ret = ivpu_hws_cmdq_init(file_priv, cmdq, VPU_ENGINE_COMPUTE, cmdq->priority); if (ret) return ret; } @@ -185,23 +231,19 @@ ivpu_cmdq_init(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq, u8 prio if (ret) return ret; - cmdq->db_registered = true; - return 0; } -static int ivpu_cmdq_fini(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) +static int ivpu_cmdq_unregister(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) { struct ivpu_device *vdev = file_priv->vdev; int ret; lockdep_assert_held(&file_priv->lock); - if (!cmdq->db_registered) + if (!cmdq->db_id) return 0; - cmdq->db_registered = false; - if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) { ret = ivpu_jsm_hws_destroy_cmdq(vdev, file_priv->ctx.id, cmdq->id); if (!ret) @@ -212,91 +254,61 @@ static int ivpu_cmdq_fini(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cm if (!ret) ivpu_dbg(vdev, JOB, "DB %d unregistered\n", cmdq->db_id); + xa_erase(&file_priv->vdev->db_xa, cmdq->db_id); + cmdq->db_id = 0; + return 0; } -static int ivpu_db_id_alloc(struct ivpu_device *vdev, u32 *db_id) +static inline u8 ivpu_job_to_jsm_priority(u8 priority) { - int ret; - u32 id; - - ret = xa_alloc_cyclic(&vdev->db_xa, &id, NULL, vdev->db_limit, &vdev->db_next, GFP_KERNEL); - if (ret < 0) - return ret; + if (priority == DRM_IVPU_JOB_PRIORITY_DEFAULT) + return VPU_JOB_SCHEDULING_PRIORITY_BAND_NORMAL; - *db_id = id; - return 0; + return priority - 1; } -static int ivpu_cmdq_id_alloc(struct ivpu_file_priv *file_priv, u32 *cmdq_id) +static void ivpu_cmdq_destroy(struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) { - int ret; - u32 id; - - ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &id, NULL, file_priv->cmdq_limit, - &file_priv->cmdq_id_next, GFP_KERNEL); - if (ret < 0) - return ret; - - *cmdq_id = id; - return 0; + ivpu_cmdq_unregister(file_priv, cmdq); + xa_erase(&file_priv->cmdq_xa, cmdq->id); + ivpu_cmdq_free(file_priv, cmdq); } -static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u8 priority) +static struct ivpu_cmdq *ivpu_cmdq_acquire_legacy(struct ivpu_file_priv *file_priv, u8 priority) { - struct ivpu_device *vdev = file_priv->vdev; struct ivpu_cmdq *cmdq; unsigned long id; - int ret; lockdep_assert_held(&file_priv->lock); xa_for_each(&file_priv->cmdq_xa, id, cmdq) - if (cmdq->priority == priority) + if (cmdq->is_legacy && cmdq->priority == priority) break; if (!cmdq) { - cmdq = ivpu_cmdq_alloc(file_priv); - if (!cmdq) { - ivpu_err(vdev, "Failed to allocate command queue\n"); + cmdq = ivpu_cmdq_create(file_priv, priority, true); + if (!cmdq) return NULL; - } + } - ret = ivpu_db_id_alloc(vdev, &cmdq->db_id); - if (ret) { - ivpu_err(file_priv->vdev, "Failed to allocate doorbell ID: %d\n", ret); - goto err_free_cmdq; - } + return cmdq; +} - ret = ivpu_cmdq_id_alloc(file_priv, &cmdq->id); - if (ret) { - ivpu_err(vdev, "Failed to allocate command queue ID: %d\n", ret); - goto err_erase_db_id; - } +static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u32 cmdq_id) +{ + struct ivpu_device *vdev = file_priv->vdev; + struct ivpu_cmdq *cmdq; - cmdq->priority = priority; - ret = xa_err(xa_store(&file_priv->cmdq_xa, cmdq->id, cmdq, GFP_KERNEL)); - if (ret) { - ivpu_err(vdev, "Failed to store command queue in cmdq_xa: %d\n", ret); - goto err_erase_cmdq_id; - } - } + lockdep_assert_held(&file_priv->lock); - ret = ivpu_cmdq_init(file_priv, cmdq, priority); - if (ret) { - ivpu_err(vdev, "Failed to initialize command queue: %d\n", ret); - goto err_free_cmdq; + cmdq = xa_load(&file_priv->cmdq_xa, cmdq_id); + if (!cmdq) { + ivpu_err(vdev, "Failed to find command queue with ID: %u\n", cmdq_id); + return NULL; } return cmdq; - -err_erase_cmdq_id: - xa_erase(&file_priv->cmdq_xa, cmdq->id); -err_erase_db_id: - xa_erase(&vdev->db_xa, cmdq->db_id); -err_free_cmdq: - ivpu_cmdq_free(file_priv, cmdq); - return NULL; } void ivpu_cmdq_release_all_locked(struct ivpu_file_priv *file_priv) @@ -306,11 +318,8 @@ void ivpu_cmdq_release_all_locked(struct ivpu_file_priv *file_priv) lockdep_assert_held(&file_priv->lock); - xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) { - xa_erase(&file_priv->cmdq_xa, cmdq_id); - ivpu_cmdq_fini(file_priv, cmdq); - ivpu_cmdq_free(file_priv, cmdq); - } + xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) + ivpu_cmdq_destroy(file_priv, cmdq); } /* @@ -326,8 +335,10 @@ static void ivpu_cmdq_reset(struct ivpu_file_priv *file_priv) mutex_lock(&file_priv->lock); - xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) - cmdq->db_registered = false; + xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) { + xa_erase(&file_priv->vdev->db_xa, cmdq->db_id); + cmdq->db_id = 0; + } mutex_unlock(&file_priv->lock); } @@ -345,13 +356,13 @@ void ivpu_cmdq_reset_all_contexts(struct ivpu_device *vdev) mutex_unlock(&vdev->context_list_lock); } -static void ivpu_cmdq_fini_all(struct ivpu_file_priv *file_priv) +static void ivpu_cmdq_unregister_all(struct ivpu_file_priv *file_priv) { struct ivpu_cmdq *cmdq; unsigned long cmdq_id; xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) - ivpu_cmdq_fini(file_priv, cmdq); + ivpu_cmdq_unregister(file_priv, cmdq); } void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv) @@ -360,7 +371,7 @@ void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv) lockdep_assert_held(&file_priv->lock); - ivpu_cmdq_fini_all(file_priv); + ivpu_cmdq_unregister_all(file_priv); if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_OS) ivpu_jsm_context_release(vdev, file_priv->ctx.id); @@ -549,7 +560,7 @@ void ivpu_jobs_abort_all(struct ivpu_device *vdev) ivpu_job_signal_and_destroy(vdev, id, DRM_IVPU_JOB_STATUS_ABORTED); } -static int ivpu_job_submit(struct ivpu_job *job, u8 priority) +static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) { struct ivpu_file_priv *file_priv = job->file_priv; struct ivpu_device *vdev = job->vdev; @@ -563,14 +574,22 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority) mutex_lock(&file_priv->lock); - cmdq = ivpu_cmdq_acquire(file_priv, priority); + if (cmdq_id == 0) + cmdq = ivpu_cmdq_acquire_legacy(file_priv, priority); + else + cmdq = ivpu_cmdq_acquire(file_priv, cmdq_id); if (!cmdq) { - ivpu_warn_ratelimited(vdev, "Failed to get job queue, ctx %d engine %d prio %d\n", - file_priv->ctx.id, job->engine_idx, priority); + ivpu_warn_ratelimited(vdev, "Failed to get job queue, ctx %d\n", file_priv->ctx.id); ret = -EINVAL; goto err_unlock_file_priv; } + ret = ivpu_cmdq_register(file_priv, cmdq); + if (ret) { + ivpu_err(vdev, "Failed to register command queue: %d\n", ret); + goto err_unlock_file_priv; + } + xa_lock(&vdev->submitted_jobs_xa); is_first_job = xa_empty(&vdev->submitted_jobs_xa); ret = __xa_alloc_cyclic(&vdev->submitted_jobs_xa, &job->job_id, job, file_priv->job_limit, @@ -599,7 +618,7 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority) trace_job("submit", job); ivpu_dbg(vdev, JOB, "Job submitted: id %3u ctx %2d engine %d prio %d addr 0x%llx next %d\n", - job->job_id, file_priv->ctx.id, job->engine_idx, priority, + job->job_id, file_priv->ctx.id, job->engine_idx, cmdq->priority, job->cmd_buf_vpu_addr, cmdq->jobq->header.tail); xa_unlock(&vdev->submitted_jobs_xa); @@ -625,7 +644,7 @@ static int ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32 *buf_handles, u32 buf_count, u32 commands_offset) { - struct ivpu_file_priv *file_priv = file->driver_priv; + struct ivpu_file_priv *file_priv = job->file_priv; struct ivpu_device *vdev = file_priv->vdev; struct ww_acquire_ctx acquire_ctx; enum dma_resv_usage usage; @@ -687,49 +706,20 @@ ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32 return ret; } -static inline u8 ivpu_job_to_hws_priority(struct ivpu_file_priv *file_priv, u8 priority) +static int ivpu_submit(struct drm_file *file, struct ivpu_file_priv *file_priv, u32 cmdq_id, + u32 buffer_count, u32 engine, void __user *buffers_ptr, u32 cmds_offset, + u8 priority) { - if (priority == DRM_IVPU_JOB_PRIORITY_DEFAULT) - return DRM_IVPU_JOB_PRIORITY_NORMAL; - - return priority - 1; -} - -int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) -{ - struct ivpu_file_priv *file_priv = file->driver_priv; struct ivpu_device *vdev = file_priv->vdev; - struct drm_ivpu_submit *params = data; struct ivpu_job *job; u32 *buf_handles; int idx, ret; - u8 priority; - - if (params->engine != DRM_IVPU_ENGINE_COMPUTE) - return -EINVAL; - - if (params->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) - return -EINVAL; - - if (params->buffer_count == 0 || params->buffer_count > JOB_MAX_BUFFER_COUNT) - return -EINVAL; - - if (!IS_ALIGNED(params->commands_offset, 8)) - return -EINVAL; - if (!file_priv->ctx.id) - return -EINVAL; - - if (file_priv->has_mmu_faults) - return -EBADFD; - - buf_handles = kcalloc(params->buffer_count, sizeof(u32), GFP_KERNEL); + buf_handles = kcalloc(buffer_count, sizeof(u32), GFP_KERNEL); if (!buf_handles) return -ENOMEM; - ret = copy_from_user(buf_handles, - (void __user *)params->buffers_ptr, - params->buffer_count * sizeof(u32)); + ret = copy_from_user(buf_handles, buffers_ptr, buffer_count * sizeof(u32)); if (ret) { ret = -EFAULT; goto err_free_handles; @@ -740,27 +730,23 @@ int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto err_free_handles; } - ivpu_dbg(vdev, JOB, "Submit ioctl: ctx %u buf_count %u\n", - file_priv->ctx.id, params->buffer_count); + ivpu_dbg(vdev, JOB, "Submit ioctl: ctx %u buf_count %u\n", file_priv->ctx.id, buffer_count); - job = ivpu_job_create(file_priv, params->engine, params->buffer_count); + job = ivpu_job_create(file_priv, engine, buffer_count); if (!job) { ivpu_err(vdev, "Failed to create job\n"); ret = -ENOMEM; goto err_exit_dev; } - ret = ivpu_job_prepare_bos_for_submit(file, job, buf_handles, params->buffer_count, - params->commands_offset); + ret = ivpu_job_prepare_bos_for_submit(file, job, buf_handles, buffer_count, cmds_offset); if (ret) { ivpu_err(vdev, "Failed to prepare job: %d\n", ret); goto err_destroy_job; } - priority = ivpu_job_to_hws_priority(file_priv, params->priority); - down_read(&vdev->pm->reset_lock); - ret = ivpu_job_submit(job, priority); + ret = ivpu_job_submit(job, priority, cmdq_id); up_read(&vdev->pm->reset_lock); if (ret) goto err_signal_fence; @@ -780,6 +766,101 @@ int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) return ret; } +int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) +{ + struct ivpu_file_priv *file_priv = file->driver_priv; + struct drm_ivpu_submit *args = data; + u8 priority; + + if (args->engine != DRM_IVPU_ENGINE_COMPUTE) + return -EINVAL; + + if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) + return -EINVAL; + + if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) + return -EINVAL; + + if (!IS_ALIGNED(args->commands_offset, 8)) + return -EINVAL; + + if (!file_priv->ctx.id) + return -EINVAL; + + if (file_priv->has_mmu_faults) + return -EBADFD; + + priority = ivpu_job_to_jsm_priority(args->priority); + + return ivpu_submit(file, file_priv, 0, args->buffer_count, args->engine, + (void __user *)args->buffers_ptr, args->commands_offset, priority); +} + +int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) +{ + struct ivpu_file_priv *file_priv = file->driver_priv; + struct drm_ivpu_cmdq_submit *args = data; + + if (args->cmdq_id < IVPU_CMDQ_MIN_ID || args->cmdq_id > IVPU_CMDQ_MAX_ID) + return -EINVAL; + + if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) + return -EINVAL; + + if (!IS_ALIGNED(args->commands_offset, 8)) + return -EINVAL; + + if (!file_priv->ctx.id) + return -EINVAL; + + if (file_priv->has_mmu_faults) + return -EBADFD; + + return ivpu_submit(file, file_priv, args->cmdq_id, args->buffer_count, VPU_ENGINE_COMPUTE, + (void __user *)args->buffers_ptr, args->commands_offset, 0); +} + +int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file) +{ + struct ivpu_file_priv *file_priv = file->driver_priv; + struct drm_ivpu_cmdq_create *args = data; + struct ivpu_cmdq *cmdq; + + if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) + return -EINVAL; + + mutex_lock(&file_priv->lock); + + cmdq = ivpu_cmdq_create(file_priv, ivpu_job_to_jsm_priority(args->priority), false); + if (cmdq) + args->cmdq_id = cmdq->id; + + mutex_unlock(&file_priv->lock); + + return cmdq ? 0 : -ENOMEM; +} + +int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file *file) +{ + struct ivpu_file_priv *file_priv = file->driver_priv; + struct drm_ivpu_cmdq_destroy *args = data; + struct ivpu_cmdq *cmdq; + int ret = 0; + + mutex_lock(&file_priv->lock); + + cmdq = xa_load(&file_priv->cmdq_xa, args->cmdq_id); + if (!cmdq || cmdq->is_legacy) { + ret = -ENOENT; + goto unlock; + } + + ivpu_cmdq_destroy(file_priv, cmdq); +unlock: + mutex_unlock(&file_priv->lock); + return ret; +} + static void ivpu_job_done_callback(struct ivpu_device *vdev, struct ivpu_ipc_hdr *ipc_hdr, struct vpu_jsm_msg *jsm_msg) diff --git a/drivers/accel/ivpu/ivpu_job.h b/drivers/accel/ivpu/ivpu_job.h index 8b19e3f8b4cf..4d31277bcc41 100644 --- a/drivers/accel/ivpu/ivpu_job.h +++ b/drivers/accel/ivpu/ivpu_job.h @@ -30,8 +30,8 @@ struct ivpu_cmdq { u32 entry_count; u32 id; u32 db_id; - bool db_registered; u8 priority; + bool is_legacy; }; /** @@ -58,6 +58,9 @@ struct ivpu_job { }; int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file); +int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file); +int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file *file); +int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file); void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv); diff --git a/include/uapi/drm/ivpu_accel.h b/include/uapi/drm/ivpu_accel.h index a35b97b097bf..746c43bd3eb6 100644 --- a/include/uapi/drm/ivpu_accel.h +++ b/include/uapi/drm/ivpu_accel.h @@ -22,6 +22,9 @@ extern "C" { #define DRM_IVPU_METRIC_STREAMER_STOP 0x08 #define DRM_IVPU_METRIC_STREAMER_GET_DATA 0x09 #define DRM_IVPU_METRIC_STREAMER_GET_INFO 0x0a +#define DRM_IVPU_CMDQ_CREATE 0x0b +#define DRM_IVPU_CMDQ_DESTROY 0x0c +#define DRM_IVPU_CMDQ_SUBMIT 0x0d #define DRM_IOCTL_IVPU_GET_PARAM \ DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_GET_PARAM, struct drm_ivpu_param) @@ -57,6 +60,15 @@ extern "C" { DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_METRIC_STREAMER_GET_INFO, \ struct drm_ivpu_metric_streamer_get_data) +#define DRM_IOCTL_IVPU_CMDQ_CREATE \ + DRM_IOWR(DRM_COMMAND_BASE + DRM_IVPU_CMDQ_CREATE, struct drm_ivpu_cmdq_create) + +#define DRM_IOCTL_IVPU_CMDQ_DESTROY \ + DRM_IOW(DRM_COMMAND_BASE + DRM_IVPU_CMDQ_DESTROY, struct drm_ivpu_cmdq_destroy) + +#define DRM_IOCTL_IVPU_CMDQ_SUBMIT \ + DRM_IOW(DRM_COMMAND_BASE + DRM_IVPU_CMDQ_SUBMIT, struct drm_ivpu_cmdq_submit) + /** * DOC: contexts * @@ -107,6 +119,13 @@ extern "C" { * accessible by hardware DMA. */ #define DRM_IVPU_CAP_DMA_MEMORY_RANGE 2 +/** + * DRM_IVPU_CAP_MANAGE_CMDQ + * + * Driver supports explicit command queue operations like command queue create, + * command queue destroy and submit job on specific command queue. + */ +#define DRM_IVPU_CAP_MANAGE_CMDQ 3 /** * struct drm_ivpu_param - Get/Set VPU parameters @@ -316,6 +335,44 @@ struct drm_ivpu_submit { __u32 priority; }; +/** + * struct drm_ivpu_cmdq_submit - Submit commands to the VPU using explicit command queue + * + * Execute a single command buffer on a given command queue. + * Handles to all referenced buffer objects have to be provided in @buffers_ptr. + * + * User space may wait on job completion using %DRM_IVPU_BO_WAIT ioctl. + */ +struct drm_ivpu_cmdq_submit { + /** + * @buffers_ptr: + * + * A pointer to an u32 array of GEM handles of the BOs required for this job. + * The number of elements in the array must be equal to the value given by @buffer_count. + * + * The first BO is the command buffer. The rest of array has to contain all + * BOs referenced from the command buffer. + */ + __u64 buffers_ptr; + + /** @buffer_count: Number of elements in the @buffers_ptr */ + __u32 buffer_count; + + /** @cmdq_id: ID for the command queue where job will be submitted */ + __u32 cmdq_id; + + /** @flags: Reserved for future use - must be zero */ + __u32 flags; + + /** + * @commands_offset: + * + * Offset inside the first buffer in @buffers_ptr containing commands + * to be executed. The offset has to be 8-byte aligned. + */ + __u32 commands_offset; +}; + /* drm_ivpu_bo_wait job status codes */ #define DRM_IVPU_JOB_STATUS_SUCCESS 0 #define DRM_IVPU_JOB_STATUS_ABORTED 256 @@ -388,6 +445,33 @@ struct drm_ivpu_metric_streamer_get_data { __u64 data_size; }; +/** + * struct drm_ivpu_cmdq_create - Create command queue for job submission + */ +struct drm_ivpu_cmdq_create { + /** @cmdq_id: Returned ID of created command queue */ + __u32 cmdq_id; + /** + * @priority: + * + * Priority to be set for related job command queue, can be one of the following: + * %DRM_IVPU_JOB_PRIORITY_DEFAULT + * %DRM_IVPU_JOB_PRIORITY_IDLE + * %DRM_IVPU_JOB_PRIORITY_NORMAL + * %DRM_IVPU_JOB_PRIORITY_FOCUS + * %DRM_IVPU_JOB_PRIORITY_REALTIME + */ + __u32 priority; +}; + +/** + * struct drm_ivpu_cmdq_destroy - Destroy a command queue + */ +struct drm_ivpu_cmdq_destroy { + /** @cmdq_id: ID of command queue to destroy */ + __u32 cmdq_id; +}; + /** * struct drm_ivpu_metric_streamer_stop - Stop collecting metric data */ From patchwork Tue Jan 7 17:32:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B10A5E7719A for ; Tue, 7 Jan 2025 14:22:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1CFDE10E6FD; Tue, 7 Jan 2025 14:22:36 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="MIPm0yvv"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id C34E410E6FB for ; Tue, 7 Jan 2025 14:22:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259754; x=1767795754; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Dw+J5EKdfxsJprPLAz2OkutidWFyShFObZUKgq92FSE=; b=MIPm0yvvTHcWD74vjQeD8KFRQKDAcIPR5NBmW166BAqq6qlwZy57/Dvi hRb7abZfIVI25xCkmmRd8XzDA3nZgzTo7ByzRsxlM7CamGUC5h5N7m6+3 uFQ42lN9uNxAkLmecc0yT9xAIqOVo62NTcYSP+WH0Yh5yWtHJphz+d+HA oC6L3x9Y5EszAzpmGbEVKq4wfmlQXCCc3tB7OYIWNzBjkYe5Jl3/+EXIL huduphuivk/h8rwep9T+otWY5xtSs88WUXZwLkyHjnr1B2xNMHNM6bE3c 1F7gzWVlRt+13zLOjH4UcQz6LbvGvIN17+QDmG4ODL5rebqrWWQuWw9sh Q==; X-CSE-ConnectionGUID: iiyzDXwZRGuglCCStg18jg== X-CSE-MsgGUID: aniSzy4pQGCdDm0i9kVfOA== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324428" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324428" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:34 -0800 X-CSE-ConnectionGUID: MI6/VC9aR76qamsy6Oz39g== X-CSE-MsgGUID: /Zreft0wSE6Tu/or14J+Bw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635419" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:33 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 03/14] accel/ivpu: Abort all jobs after command queue unregister Date: Tue, 7 Jan 2025 18:32:26 +0100 Message-ID: <20250107173238.381120-4-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski With hardware scheduler it is not expected to receive JOB_DONE notifications from NPU FW for the jobs aborted due to command queue destroy JSM command. Remove jobs submitted to unregistered command queue from submitted_jobs_xa to avoid triggering a TDR in such case. Add explicit submitted_jobs_lock that protects access to list of submitted jobs which is now used to find jobs to abort. Move context abort procedure to separate work queue not to slow down handling of IPCs or DCT requests in case where job abort takes longer, especially when destruction of the last job of a specific context results in context release. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_drv.c | 32 ++------ drivers/accel/ivpu/ivpu_drv.h | 2 + drivers/accel/ivpu/ivpu_job.c | 137 ++++++++++++++++++++++++-------- drivers/accel/ivpu/ivpu_job.h | 4 + drivers/accel/ivpu/ivpu_mmu.c | 3 +- drivers/accel/ivpu/ivpu_sysfs.c | 5 +- 6 files changed, 121 insertions(+), 62 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_drv.c b/drivers/accel/ivpu/ivpu_drv.c index f4171536640b..300eea8c305f 100644 --- a/drivers/accel/ivpu/ivpu_drv.c +++ b/drivers/accel/ivpu/ivpu_drv.c @@ -36,8 +36,6 @@ #define DRIVER_VERSION_STR "1.0.0 " UTS_RELEASE #endif -static struct lock_class_key submitted_jobs_xa_lock_class_key; - int ivpu_dbg_mask; module_param_named(dbg_mask, ivpu_dbg_mask, int, 0644); MODULE_PARM_DESC(dbg_mask, "Driver debug mask. See IVPU_DBG_* macros."); @@ -467,26 +465,6 @@ static const struct drm_driver driver = { .major = 1, }; -static void ivpu_context_abort_invalid(struct ivpu_device *vdev) -{ - struct ivpu_file_priv *file_priv; - unsigned long ctx_id; - - mutex_lock(&vdev->context_list_lock); - - xa_for_each(&vdev->context_xa, ctx_id, file_priv) { - if (!file_priv->has_mmu_faults || file_priv->aborted) - continue; - - mutex_lock(&file_priv->lock); - ivpu_context_abort_locked(file_priv); - file_priv->aborted = true; - mutex_unlock(&file_priv->lock); - } - - mutex_unlock(&vdev->context_list_lock); -} - static irqreturn_t ivpu_irq_thread_handler(int irq, void *arg) { struct ivpu_device *vdev = arg; @@ -500,9 +478,6 @@ static irqreturn_t ivpu_irq_thread_handler(int irq, void *arg) case IVPU_HW_IRQ_SRC_IPC: ivpu_ipc_irq_thread_handler(vdev); break; - case IVPU_HW_IRQ_SRC_MMU_EVTQ: - ivpu_context_abort_invalid(vdev); - break; case IVPU_HW_IRQ_SRC_DCT: ivpu_pm_dct_irq_thread_handler(vdev); break; @@ -619,16 +594,21 @@ static int ivpu_dev_init(struct ivpu_device *vdev) xa_init_flags(&vdev->context_xa, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ); xa_init_flags(&vdev->submitted_jobs_xa, XA_FLAGS_ALLOC1); xa_init_flags(&vdev->db_xa, XA_FLAGS_ALLOC1); - lockdep_set_class(&vdev->submitted_jobs_xa.xa_lock, &submitted_jobs_xa_lock_class_key); INIT_LIST_HEAD(&vdev->bo_list); vdev->db_limit.min = IVPU_MIN_DB; vdev->db_limit.max = IVPU_MAX_DB; + INIT_WORK(&vdev->context_abort_work, ivpu_context_abort_thread_handler); + ret = drmm_mutex_init(&vdev->drm, &vdev->context_list_lock); if (ret) goto err_xa_destroy; + ret = drmm_mutex_init(&vdev->drm, &vdev->submitted_jobs_lock); + if (ret) + goto err_xa_destroy; + ret = drmm_mutex_init(&vdev->drm, &vdev->bo_list_lock); if (ret) goto err_xa_destroy; diff --git a/drivers/accel/ivpu/ivpu_drv.h b/drivers/accel/ivpu/ivpu_drv.h index 3fdff3f6cffd..ebfcf3e42a3d 100644 --- a/drivers/accel/ivpu/ivpu_drv.h +++ b/drivers/accel/ivpu/ivpu_drv.h @@ -137,6 +137,7 @@ struct ivpu_device { struct mutex context_list_lock; /* Protects user context addition/removal */ struct xarray context_xa; struct xa_limit context_xa_limit; + struct work_struct context_abort_work; struct xarray db_xa; struct xa_limit db_limit; @@ -145,6 +146,7 @@ struct ivpu_device { struct mutex bo_list_lock; /* Protects bo_list */ struct list_head bo_list; + struct mutex submitted_jobs_lock; /* Protects submitted_jobs */ struct xarray submitted_jobs_xa; struct ivpu_ipc_consumer job_done_consumer; diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index 43507d3aea51..7fed3c8406ee 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -122,6 +122,7 @@ static struct ivpu_cmdq *ivpu_cmdq_create(struct ivpu_file_priv *file_priv, u8 p cmdq->priority = priority; cmdq->is_legacy = is_legacy; + cmdq->is_valid = true; ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &cmdq->id, cmdq, file_priv->cmdq_limit, &file_priv->cmdq_id_next, GFP_KERNEL); @@ -130,6 +131,7 @@ static struct ivpu_cmdq *ivpu_cmdq_create(struct ivpu_file_priv *file_priv, u8 p goto err_free_cmdq; } + ivpu_dbg(vdev, JOB, "Command queue %d created, ctx %d\n", cmdq->id, file_priv->ctx.id); return cmdq; err_free_cmdq: @@ -247,7 +249,8 @@ static int ivpu_cmdq_unregister(struct ivpu_file_priv *file_priv, struct ivpu_cm if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) { ret = ivpu_jsm_hws_destroy_cmdq(vdev, file_priv->ctx.id, cmdq->id); if (!ret) - ivpu_dbg(vdev, JOB, "Command queue %d destroyed\n", cmdq->id); + ivpu_dbg(vdev, JOB, "Command queue %d destroyed, ctx %d\n", + cmdq->id, file_priv->ctx.id); } ret = ivpu_jsm_unregister_db(vdev, cmdq->db_id); @@ -303,8 +306,8 @@ static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u32 lockdep_assert_held(&file_priv->lock); cmdq = xa_load(&file_priv->cmdq_xa, cmdq_id); - if (!cmdq) { - ivpu_err(vdev, "Failed to find command queue with ID: %u\n", cmdq_id); + if (!cmdq || !cmdq->is_valid) { + ivpu_warn_ratelimited(vdev, "Failed to find command queue with ID: %u\n", cmdq_id); return NULL; } @@ -356,25 +359,21 @@ void ivpu_cmdq_reset_all_contexts(struct ivpu_device *vdev) mutex_unlock(&vdev->context_list_lock); } -static void ivpu_cmdq_unregister_all(struct ivpu_file_priv *file_priv) -{ - struct ivpu_cmdq *cmdq; - unsigned long cmdq_id; - - xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) - ivpu_cmdq_unregister(file_priv, cmdq); -} - void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv) { struct ivpu_device *vdev = file_priv->vdev; + struct ivpu_cmdq *cmdq; + unsigned long cmdq_id; lockdep_assert_held(&file_priv->lock); - ivpu_cmdq_unregister_all(file_priv); + xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) + ivpu_cmdq_unregister(file_priv, cmdq); if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_OS) ivpu_jsm_context_release(vdev, file_priv->ctx.id); + + file_priv->aborted = true; } static int ivpu_cmdq_push_job(struct ivpu_cmdq *cmdq, struct ivpu_job *job) @@ -513,16 +512,14 @@ static struct ivpu_job *ivpu_job_remove_from_submitted_jobs(struct ivpu_device * { struct ivpu_job *job; - xa_lock(&vdev->submitted_jobs_xa); - job = __xa_erase(&vdev->submitted_jobs_xa, job_id); + lockdep_assert_held(&vdev->submitted_jobs_lock); + job = xa_erase(&vdev->submitted_jobs_xa, job_id); if (xa_empty(&vdev->submitted_jobs_xa) && job) { vdev->busy_time = ktime_add(ktime_sub(ktime_get(), vdev->busy_start_ts), vdev->busy_time); } - xa_unlock(&vdev->submitted_jobs_xa); - return job; } @@ -530,6 +527,8 @@ static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 { struct ivpu_job *job; + lockdep_assert_held(&vdev->submitted_jobs_lock); + job = ivpu_job_remove_from_submitted_jobs(vdev, job_id); if (!job) return -ENOENT; @@ -548,6 +547,10 @@ static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 ivpu_stop_job_timeout_detection(vdev); ivpu_rpm_put(vdev); + + if (!xa_empty(&vdev->submitted_jobs_xa)) + ivpu_start_job_timeout_detection(vdev); + return 0; } @@ -556,8 +559,26 @@ void ivpu_jobs_abort_all(struct ivpu_device *vdev) struct ivpu_job *job; unsigned long id; + mutex_lock(&vdev->submitted_jobs_lock); + xa_for_each(&vdev->submitted_jobs_xa, id, job) ivpu_job_signal_and_destroy(vdev, id, DRM_IVPU_JOB_STATUS_ABORTED); + + mutex_unlock(&vdev->submitted_jobs_lock); +} + +void ivpu_cmdq_abort_all_jobs(struct ivpu_device *vdev, u32 ctx_id, u32 cmdq_id) +{ + struct ivpu_job *job; + unsigned long id; + + mutex_lock(&vdev->submitted_jobs_lock); + + xa_for_each(&vdev->submitted_jobs_xa, id, job) + if (job->file_priv->ctx.id == ctx_id && job->cmdq_id == cmdq_id) + ivpu_job_signal_and_destroy(vdev, id, DRM_IVPU_JOB_STATUS_ABORTED); + + mutex_unlock(&vdev->submitted_jobs_lock); } static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) @@ -590,15 +611,18 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) goto err_unlock_file_priv; } - xa_lock(&vdev->submitted_jobs_xa); + job->cmdq_id = cmdq->id; + + mutex_lock(&vdev->submitted_jobs_lock); + is_first_job = xa_empty(&vdev->submitted_jobs_xa); - ret = __xa_alloc_cyclic(&vdev->submitted_jobs_xa, &job->job_id, job, file_priv->job_limit, - &file_priv->job_id_next, GFP_KERNEL); + ret = xa_alloc_cyclic(&vdev->submitted_jobs_xa, &job->job_id, job, file_priv->job_limit, + &file_priv->job_id_next, GFP_KERNEL); if (ret < 0) { ivpu_dbg(vdev, JOB, "Too many active jobs in ctx %d\n", file_priv->ctx.id); ret = -EBUSY; - goto err_unlock_submitted_jobs_xa; + goto err_unlock_submitted_jobs; } ret = ivpu_cmdq_push_job(cmdq, job); @@ -621,19 +645,21 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) job->job_id, file_priv->ctx.id, job->engine_idx, cmdq->priority, job->cmd_buf_vpu_addr, cmdq->jobq->header.tail); - xa_unlock(&vdev->submitted_jobs_xa); - + mutex_unlock(&vdev->submitted_jobs_lock); mutex_unlock(&file_priv->lock); - if (unlikely(ivpu_test_mode & IVPU_TEST_MODE_NULL_HW)) + if (unlikely(ivpu_test_mode & IVPU_TEST_MODE_NULL_HW)) { + mutex_lock(&vdev->submitted_jobs_lock); ivpu_job_signal_and_destroy(vdev, job->job_id, VPU_JSM_STATUS_SUCCESS); + mutex_unlock(&vdev->submitted_jobs_lock); + } return 0; err_erase_xa: - __xa_erase(&vdev->submitted_jobs_xa, job->job_id); -err_unlock_submitted_jobs_xa: - xa_unlock(&vdev->submitted_jobs_xa); + xa_erase(&vdev->submitted_jobs_xa, job->job_id); +err_unlock_submitted_jobs: + mutex_unlock(&vdev->submitted_jobs_lock); err_unlock_file_priv: mutex_unlock(&file_priv->lock); ivpu_rpm_put(vdev); @@ -843,19 +869,33 @@ int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file * int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file *file) { struct ivpu_file_priv *file_priv = file->driver_priv; + struct ivpu_device *vdev = file_priv->vdev; struct drm_ivpu_cmdq_destroy *args = data; struct ivpu_cmdq *cmdq; + u32 cmdq_id; int ret = 0; mutex_lock(&file_priv->lock); cmdq = xa_load(&file_priv->cmdq_xa, args->cmdq_id); - if (!cmdq || cmdq->is_legacy) { + if (!cmdq || !cmdq->is_valid || cmdq->is_legacy) { ret = -ENOENT; goto unlock; } + /* + * There is no way to stop executing jobs per command queue + * in OS scheduling mode, mark command queue as invalid instead + * and it will be freed together with context release. + */ + if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_OS) { + cmdq->is_valid = false; + goto unlock; + } + + cmdq_id = cmdq->id; ivpu_cmdq_destroy(file_priv, cmdq); + ivpu_cmdq_abort_all_jobs(vdev, file_priv->ctx.id, cmdq_id); unlock: mutex_unlock(&file_priv->lock); return ret; @@ -866,7 +906,6 @@ ivpu_job_done_callback(struct ivpu_device *vdev, struct ivpu_ipc_hdr *ipc_hdr, struct vpu_jsm_msg *jsm_msg) { struct vpu_ipc_msg_payload_job_done *payload; - int ret; if (!jsm_msg) { ivpu_err(vdev, "IPC message has no JSM payload\n"); @@ -879,9 +918,10 @@ ivpu_job_done_callback(struct ivpu_device *vdev, struct ivpu_ipc_hdr *ipc_hdr, } payload = (struct vpu_ipc_msg_payload_job_done *)&jsm_msg->payload; - ret = ivpu_job_signal_and_destroy(vdev, payload->job_id, payload->job_status); - if (!ret && !xa_empty(&vdev->submitted_jobs_xa)) - ivpu_start_job_timeout_detection(vdev); + + mutex_lock(&vdev->submitted_jobs_lock); + ivpu_job_signal_and_destroy(vdev, payload->job_id, payload->job_status); + mutex_unlock(&vdev->submitted_jobs_lock); } void ivpu_job_done_consumer_init(struct ivpu_device *vdev) @@ -894,3 +934,36 @@ void ivpu_job_done_consumer_fini(struct ivpu_device *vdev) { ivpu_ipc_consumer_del(vdev, &vdev->job_done_consumer); } + +void ivpu_context_abort_thread_handler(struct work_struct *work) +{ + struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); + struct ivpu_file_priv *file_priv; + unsigned long ctx_id; + struct ivpu_job *job; + unsigned long id; + + mutex_lock(&vdev->context_list_lock); + xa_for_each(&vdev->context_xa, ctx_id, file_priv) { + if (!file_priv->has_mmu_faults || file_priv->aborted) + continue; + + mutex_lock(&file_priv->lock); + ivpu_context_abort_locked(file_priv); + mutex_unlock(&file_priv->lock); + } + mutex_unlock(&vdev->context_list_lock); + + if (vdev->fw->sched_mode != VPU_SCHEDULING_MODE_HW) + return; + /* + * In hardware scheduling mode NPU already has stopped processing jobs + * and won't send us any further notifications, thus we have to free job related resources + * and notify userspace + */ + mutex_lock(&vdev->submitted_jobs_lock); + xa_for_each(&vdev->submitted_jobs_xa, id, job) + if (job->file_priv->aborted) + ivpu_job_signal_and_destroy(vdev, job->job_id, DRM_IVPU_JOB_STATUS_ABORTED); + mutex_unlock(&vdev->submitted_jobs_lock); +} diff --git a/drivers/accel/ivpu/ivpu_job.h b/drivers/accel/ivpu/ivpu_job.h index 4d31277bcc41..fef8aed1fc88 100644 --- a/drivers/accel/ivpu/ivpu_job.h +++ b/drivers/accel/ivpu/ivpu_job.h @@ -31,6 +31,7 @@ struct ivpu_cmdq { u32 id; u32 db_id; u8 priority; + bool is_valid; bool is_legacy; }; @@ -51,6 +52,7 @@ struct ivpu_job { struct ivpu_file_priv *file_priv; struct dma_fence *done_fence; u64 cmd_buf_vpu_addr; + u32 cmdq_id; u32 job_id; u32 engine_idx; size_t bo_count; @@ -66,9 +68,11 @@ void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv); void ivpu_cmdq_release_all_locked(struct ivpu_file_priv *file_priv); void ivpu_cmdq_reset_all_contexts(struct ivpu_device *vdev); +void ivpu_cmdq_abort_all_jobs(struct ivpu_device *vdev, u32 ctx_id, u32 cmdq_id); void ivpu_job_done_consumer_init(struct ivpu_device *vdev); void ivpu_job_done_consumer_fini(struct ivpu_device *vdev); +void ivpu_context_abort_thread_handler(struct work_struct *work); void ivpu_jobs_abort_all(struct ivpu_device *vdev); diff --git a/drivers/accel/ivpu/ivpu_mmu.c b/drivers/accel/ivpu/ivpu_mmu.c index 26ef52fbb93e..21f820dd0c65 100644 --- a/drivers/accel/ivpu/ivpu_mmu.c +++ b/drivers/accel/ivpu/ivpu_mmu.c @@ -890,8 +890,7 @@ void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) REGV_WR32(IVPU_MMU_REG_EVTQ_CONS_SEC, vdev->mmu->evtq.cons); } - if (!kfifo_put(&vdev->hw->irq.fifo, IVPU_HW_IRQ_SRC_MMU_EVTQ)) - ivpu_err_ratelimited(vdev, "IRQ FIFO full\n"); + queue_work(system_wq, &vdev->context_abort_work); } void ivpu_mmu_evtq_dump(struct ivpu_device *vdev) diff --git a/drivers/accel/ivpu/ivpu_sysfs.c b/drivers/accel/ivpu/ivpu_sysfs.c index 616477fc17fa..8a616791c32f 100644 --- a/drivers/accel/ivpu/ivpu_sysfs.c +++ b/drivers/accel/ivpu/ivpu_sysfs.c @@ -30,11 +30,12 @@ npu_busy_time_us_show(struct device *dev, struct device_attribute *attr, char *b struct ivpu_device *vdev = to_ivpu_device(drm); ktime_t total, now = 0; - xa_lock(&vdev->submitted_jobs_xa); + mutex_lock(&vdev->submitted_jobs_lock); + total = vdev->busy_time; if (!xa_empty(&vdev->submitted_jobs_xa)) now = ktime_sub(ktime_get(), vdev->busy_start_ts); - xa_unlock(&vdev->submitted_jobs_xa); + mutex_unlock(&vdev->submitted_jobs_lock); return sysfs_emit(buf, "%lld\n", ktime_to_us(ktime_add(total, now))); } From patchwork Tue Jan 7 17:32:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929028 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D555E77198 for ; Tue, 7 Jan 2025 14:22:38 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7C43910E700; Tue, 7 Jan 2025 14:22:37 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="eXqVM1wg"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3EC6410E700 for ; Tue, 7 Jan 2025 14:22:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259756; x=1767795756; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=m2Bkmo0oDud3Gq/Vd+Y1q9kccBCi2UZ6RyABcC7Y/Jk=; b=eXqVM1wgEgkB02wNHMECYOKUyW7WWJDhoItdDnAhdKVm5ygjz/ZSszh0 Ylq8vhj4DizWLvlWjDHrPceyoXKK3zmmEotGekmDE2GQJwVP+xtO+5tNd iJA4LmKfjCs7yEU9GkuLyKkSnbeiIOhL5HEHgesAqcaVuRgJN8d/IMsY2 dlU9nxEF6Ek/qS64sHnK95A1YedIlIX+eT5XpIXM9/9UnkYt7lvYqmBEI lLbOl5Yo9JkR+efgCGXrVtut+Dv9OnqtgIlzWKKeeuqIp5UzCmxVR2PeB OanoSi3hBBLOkOlITBRN1sjiCqygdkhh7RzR1Djg/excdQv3UibgggDWK Q==; X-CSE-ConnectionGUID: 1q22kjKhTYO1ZdYdq95r2A== X-CSE-MsgGUID: dJm+o3eoQJmeXjamVpi6aw== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324434" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324434" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:36 -0800 X-CSE-ConnectionGUID: l1+O0xvoTb+VYSX5yiNDNw== X-CSE-MsgGUID: yjHS969oSWyI4CWQ7xaF4g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635421" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:35 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Maciej Falkowski Subject: [PATCH 04/14] accel/ivpu: Expose NPU memory utilization info in sysfs Date: Tue, 7 Jan 2025 18:32:27 +0100 Message-ID: <20250107173238.381120-5-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Expose NPU memory utilization info in sysfs in bytes to show total memory used by NPU (FW + runtime). Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_sysfs.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/drivers/accel/ivpu/ivpu_sysfs.c b/drivers/accel/ivpu/ivpu_sysfs.c index 8a616791c32f..97102feaf8dd 100644 --- a/drivers/accel/ivpu/ivpu_sysfs.c +++ b/drivers/accel/ivpu/ivpu_sysfs.c @@ -7,11 +7,14 @@ #include #include "ivpu_drv.h" +#include "ivpu_gem.h" #include "ivpu_fw.h" #include "ivpu_hw.h" #include "ivpu_sysfs.h" -/* +/** + * DOC: npu_busy_time_us + * * npu_busy_time_us is the time that the device spent executing jobs. * The time is counted when and only when there are jobs submitted to firmware. * @@ -42,6 +45,30 @@ npu_busy_time_us_show(struct device *dev, struct device_attribute *attr, char *b static DEVICE_ATTR_RO(npu_busy_time_us); +/** + * DOC: npu_memory_utilization + * + * The npu_memory_utilization is used to report in bytes a current NPU memory utilization. + * + */ +static ssize_t +npu_memory_utilization_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct drm_device *drm = dev_get_drvdata(dev); + struct ivpu_device *vdev = to_ivpu_device(drm); + struct ivpu_bo *bo; + u64 total_npu_memory = 0; + + mutex_lock(&vdev->bo_list_lock); + list_for_each_entry(bo, &vdev->bo_list, bo_list_node) + total_npu_memory += bo->base.base.size; + mutex_unlock(&vdev->bo_list_lock); + + return sysfs_emit(buf, "%lld\n", total_npu_memory); +} + +static DEVICE_ATTR_RO(npu_memory_utilization); + /** * DOC: sched_mode * @@ -65,6 +92,7 @@ static DEVICE_ATTR_RO(sched_mode); static struct attribute *ivpu_dev_attrs[] = { &dev_attr_npu_busy_time_us.attr, + &dev_attr_npu_memory_utilization.attr, &dev_attr_sched_mode.attr, NULL, }; From patchwork Tue Jan 7 17:32:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FE3CE77197 for ; Tue, 7 Jan 2025 14:22:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C4E4A10E701; Tue, 7 Jan 2025 14:22:38 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="CC1CIgNg"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id EC8F310E701 for ; Tue, 7 Jan 2025 14:22:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259758; x=1767795758; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PF9DTJNSDXTHqz/slUSmRuOGFXrru5BGvCStiy2VvL4=; b=CC1CIgNgsbcc+lqS/C9z6AGO70sUok3N49b16qjTbUkDEDbtX1LxndNW lwqNuHx3j3KrdACReYFedNqEj9myUz/DQvVAXr6m2+DFsidjOku8L9dyC 8vLoTFGXEwrQ3pAIvDXPqNvEbgm4ZT6TPUceup3NPUvuI4KhxyJbQRS4r KiwpmnTJLMzV6KKFApiy/GoZfjQCAl2mRygMPNMzFNaX+dv5tgzNR0qFX UtwR/533N/VWg5z/qHQreHXsFLoufAC1v/bzI/PagvbWDjKtF+MFPw8OM rSicIw95PYq/LgVAmLO5knWN6uL+XuIueMQT3mrJLsSJLJ4yoSkEkTpvS A==; X-CSE-ConnectionGUID: AupudQMrTnqCBrV4TZhSoQ== X-CSE-MsgGUID: Z5oeot9nSCqzxywVw2q7Fg== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324438" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324438" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:37 -0800 X-CSE-ConnectionGUID: KaQPzIVSQvqe6M65jTLRUA== X-CSE-MsgGUID: zdP4Ifz5Rim7vN/K8/Ifow== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635430" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:37 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Maciej Falkowski Subject: [PATCH 05/14] accel/ivpu: Use workqueue for IRQ handling Date: Tue, 7 Jan 2025 18:32:28 +0100 Message-ID: <20250107173238.381120-6-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Convert IRQ bottom half from the thread handler into workqueue. This increases a stability in rare scenarios where driver on debugging/hardening kernels processes IRQ too slow and misses some interrupts due to it. Workqueue handler also gives a very minor performance increase. Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_drv.c | 39 ++++++++----------------------- drivers/accel/ivpu/ivpu_drv.h | 5 +++- drivers/accel/ivpu/ivpu_hw.c | 5 ---- drivers/accel/ivpu/ivpu_hw.h | 9 ------- drivers/accel/ivpu/ivpu_hw_btrs.c | 3 +-- drivers/accel/ivpu/ivpu_ipc.c | 7 +++--- drivers/accel/ivpu/ivpu_ipc.h | 2 +- drivers/accel/ivpu/ivpu_job.c | 2 +- drivers/accel/ivpu/ivpu_job.h | 2 +- drivers/accel/ivpu/ivpu_pm.c | 3 ++- drivers/accel/ivpu/ivpu_pm.h | 2 +- 11 files changed, 24 insertions(+), 55 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_drv.c b/drivers/accel/ivpu/ivpu_drv.c index 300eea8c305f..9b0d99873fb3 100644 --- a/drivers/accel/ivpu/ivpu_drv.c +++ b/drivers/accel/ivpu/ivpu_drv.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -421,6 +422,9 @@ void ivpu_prepare_for_reset(struct ivpu_device *vdev) { ivpu_hw_irq_disable(vdev); disable_irq(vdev->irq); + cancel_work_sync(&vdev->irq_ipc_work); + cancel_work_sync(&vdev->irq_dct_work); + cancel_work_sync(&vdev->context_abort_work); ivpu_ipc_disable(vdev); ivpu_mmu_disable(vdev); } @@ -465,31 +469,6 @@ static const struct drm_driver driver = { .major = 1, }; -static irqreturn_t ivpu_irq_thread_handler(int irq, void *arg) -{ - struct ivpu_device *vdev = arg; - u8 irq_src; - - if (kfifo_is_empty(&vdev->hw->irq.fifo)) - return IRQ_NONE; - - while (kfifo_get(&vdev->hw->irq.fifo, &irq_src)) { - switch (irq_src) { - case IVPU_HW_IRQ_SRC_IPC: - ivpu_ipc_irq_thread_handler(vdev); - break; - case IVPU_HW_IRQ_SRC_DCT: - ivpu_pm_dct_irq_thread_handler(vdev); - break; - default: - ivpu_err_ratelimited(vdev, "Unknown IRQ source: %u\n", irq_src); - break; - } - } - - return IRQ_HANDLED; -} - static int ivpu_irq_init(struct ivpu_device *vdev) { struct pci_dev *pdev = to_pci_dev(vdev->drm.dev); @@ -501,12 +480,16 @@ static int ivpu_irq_init(struct ivpu_device *vdev) return ret; } + INIT_WORK(&vdev->irq_ipc_work, ivpu_ipc_irq_work_fn); + INIT_WORK(&vdev->irq_dct_work, ivpu_pm_irq_dct_work_fn); + INIT_WORK(&vdev->context_abort_work, ivpu_context_abort_work_fn); + ivpu_irq_handlers_init(vdev); vdev->irq = pci_irq_vector(pdev, 0); - ret = devm_request_threaded_irq(vdev->drm.dev, vdev->irq, ivpu_hw_irq_handler, - ivpu_irq_thread_handler, IRQF_NO_AUTOEN, DRIVER_NAME, vdev); + ret = devm_request_irq(vdev->drm.dev, vdev->irq, ivpu_hw_irq_handler, + IRQF_NO_AUTOEN, DRIVER_NAME, vdev); if (ret) ivpu_err(vdev, "Failed to request an IRQ %d\n", ret); @@ -599,8 +582,6 @@ static int ivpu_dev_init(struct ivpu_device *vdev) vdev->db_limit.min = IVPU_MIN_DB; vdev->db_limit.max = IVPU_MAX_DB; - INIT_WORK(&vdev->context_abort_work, ivpu_context_abort_thread_handler); - ret = drmm_mutex_init(&vdev->drm, &vdev->context_list_lock); if (ret) goto err_xa_destroy; diff --git a/drivers/accel/ivpu/ivpu_drv.h b/drivers/accel/ivpu/ivpu_drv.h index ebfcf3e42a3d..b57d878f2fcd 100644 --- a/drivers/accel/ivpu/ivpu_drv.h +++ b/drivers/accel/ivpu/ivpu_drv.h @@ -137,12 +137,15 @@ struct ivpu_device { struct mutex context_list_lock; /* Protects user context addition/removal */ struct xarray context_xa; struct xa_limit context_xa_limit; - struct work_struct context_abort_work; struct xarray db_xa; struct xa_limit db_limit; u32 db_next; + struct work_struct irq_ipc_work; + struct work_struct irq_dct_work; + struct work_struct context_abort_work; + struct mutex bo_list_lock; /* Protects bo_list */ struct list_head bo_list; diff --git a/drivers/accel/ivpu/ivpu_hw.c b/drivers/accel/ivpu/ivpu_hw.c index 4e1054f3466e..1b691375ee4d 100644 --- a/drivers/accel/ivpu/ivpu_hw.c +++ b/drivers/accel/ivpu/ivpu_hw.c @@ -285,8 +285,6 @@ void ivpu_hw_profiling_freq_drive(struct ivpu_device *vdev, bool enable) void ivpu_irq_handlers_init(struct ivpu_device *vdev) { - INIT_KFIFO(vdev->hw->irq.fifo); - if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) vdev->hw->irq.ip_irq_handler = ivpu_hw_ip_irq_handler_37xx; else @@ -300,7 +298,6 @@ void ivpu_irq_handlers_init(struct ivpu_device *vdev) void ivpu_hw_irq_enable(struct ivpu_device *vdev) { - kfifo_reset(&vdev->hw->irq.fifo); ivpu_hw_ip_irq_enable(vdev); ivpu_hw_btrs_irq_enable(vdev); } @@ -327,8 +324,6 @@ irqreturn_t ivpu_hw_irq_handler(int irq, void *ptr) /* Re-enable global interrupts to re-trigger MSI for pending interrupts */ ivpu_hw_btrs_global_int_enable(vdev); - if (!kfifo_is_empty(&vdev->hw->irq.fifo)) - return IRQ_WAKE_THREAD; if (ip_handled || btrs_handled) return IRQ_HANDLED; return IRQ_NONE; diff --git a/drivers/accel/ivpu/ivpu_hw.h b/drivers/accel/ivpu/ivpu_hw.h index fc4dbfc980c8..fbef9816b9d0 100644 --- a/drivers/accel/ivpu/ivpu_hw.h +++ b/drivers/accel/ivpu/ivpu_hw.h @@ -6,18 +6,10 @@ #ifndef __IVPU_HW_H__ #define __IVPU_HW_H__ -#include - #include "ivpu_drv.h" #include "ivpu_hw_btrs.h" #include "ivpu_hw_ip.h" -#define IVPU_HW_IRQ_FIFO_LENGTH 1024 - -#define IVPU_HW_IRQ_SRC_IPC 1 -#define IVPU_HW_IRQ_SRC_MMU_EVTQ 2 -#define IVPU_HW_IRQ_SRC_DCT 3 - struct ivpu_addr_range { resource_size_t start; resource_size_t end; @@ -27,7 +19,6 @@ struct ivpu_hw_info { struct { bool (*btrs_irq_handler)(struct ivpu_device *vdev, int irq); bool (*ip_irq_handler)(struct ivpu_device *vdev, int irq); - DECLARE_KFIFO(fifo, u8, IVPU_HW_IRQ_FIFO_LENGTH); } irq; struct { struct ivpu_addr_range global; diff --git a/drivers/accel/ivpu/ivpu_hw_btrs.c b/drivers/accel/ivpu/ivpu_hw_btrs.c index 3212c99f3682..3753b00ed2d6 100644 --- a/drivers/accel/ivpu/ivpu_hw_btrs.c +++ b/drivers/accel/ivpu/ivpu_hw_btrs.c @@ -630,8 +630,7 @@ bool ivpu_hw_btrs_irq_handler_lnl(struct ivpu_device *vdev, int irq) if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, SURV_ERR, status)) { ivpu_dbg(vdev, IRQ, "Survivability IRQ\n"); - if (!kfifo_put(&vdev->hw->irq.fifo, IVPU_HW_IRQ_SRC_DCT)) - ivpu_err_ratelimited(vdev, "IRQ FIFO full\n"); + queue_work(system_wq, &vdev->irq_dct_work); } if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, FREQ_CHANGE, status)) diff --git a/drivers/accel/ivpu/ivpu_ipc.c b/drivers/accel/ivpu/ivpu_ipc.c index 01ebf88fe6ef..0e096fd9b95d 100644 --- a/drivers/accel/ivpu/ivpu_ipc.c +++ b/drivers/accel/ivpu/ivpu_ipc.c @@ -459,13 +459,12 @@ void ivpu_ipc_irq_handler(struct ivpu_device *vdev) } } - if (!list_empty(&ipc->cb_msg_list)) - if (!kfifo_put(&vdev->hw->irq.fifo, IVPU_HW_IRQ_SRC_IPC)) - ivpu_err_ratelimited(vdev, "IRQ FIFO full\n"); + queue_work(system_wq, &vdev->irq_ipc_work); } -void ivpu_ipc_irq_thread_handler(struct ivpu_device *vdev) +void ivpu_ipc_irq_work_fn(struct work_struct *work) { + struct ivpu_device *vdev = container_of(work, struct ivpu_device, irq_ipc_work); struct ivpu_ipc_info *ipc = vdev->ipc; struct ivpu_ipc_rx_msg *rx_msg, *r; struct list_head cb_msg_list; diff --git a/drivers/accel/ivpu/ivpu_ipc.h b/drivers/accel/ivpu/ivpu_ipc.h index b4dfb504679b..b524a1985b9d 100644 --- a/drivers/accel/ivpu/ivpu_ipc.h +++ b/drivers/accel/ivpu/ivpu_ipc.h @@ -90,7 +90,7 @@ void ivpu_ipc_disable(struct ivpu_device *vdev); void ivpu_ipc_reset(struct ivpu_device *vdev); void ivpu_ipc_irq_handler(struct ivpu_device *vdev); -void ivpu_ipc_irq_thread_handler(struct ivpu_device *vdev); +void ivpu_ipc_irq_work_fn(struct work_struct *work); void ivpu_ipc_consumer_add(struct ivpu_device *vdev, struct ivpu_ipc_consumer *cons, u32 channel, ivpu_ipc_rx_callback_t callback); diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index 7fed3c8406ee..c678dcddb8d8 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -935,7 +935,7 @@ void ivpu_job_done_consumer_fini(struct ivpu_device *vdev) ivpu_ipc_consumer_del(vdev, &vdev->job_done_consumer); } -void ivpu_context_abort_thread_handler(struct work_struct *work) +void ivpu_context_abort_work_fn(struct work_struct *work) { struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); struct ivpu_file_priv *file_priv; diff --git a/drivers/accel/ivpu/ivpu_job.h b/drivers/accel/ivpu/ivpu_job.h index fef8aed1fc88..ff77ee1fcee2 100644 --- a/drivers/accel/ivpu/ivpu_job.h +++ b/drivers/accel/ivpu/ivpu_job.h @@ -72,7 +72,7 @@ void ivpu_cmdq_abort_all_jobs(struct ivpu_device *vdev, u32 ctx_id, u32 cmdq_id) void ivpu_job_done_consumer_init(struct ivpu_device *vdev); void ivpu_job_done_consumer_fini(struct ivpu_device *vdev); -void ivpu_context_abort_thread_handler(struct work_struct *work); +void ivpu_context_abort_work_fn(struct work_struct *work); void ivpu_jobs_abort_all(struct ivpu_device *vdev); diff --git a/drivers/accel/ivpu/ivpu_pm.c b/drivers/accel/ivpu/ivpu_pm.c index 6821051dfa3a..f41b3bfe40af 100644 --- a/drivers/accel/ivpu/ivpu_pm.c +++ b/drivers/accel/ivpu/ivpu_pm.c @@ -452,8 +452,9 @@ int ivpu_pm_dct_disable(struct ivpu_device *vdev) return 0; } -void ivpu_pm_dct_irq_thread_handler(struct ivpu_device *vdev) +void ivpu_pm_irq_dct_work_fn(struct work_struct *work) { + struct ivpu_device *vdev = container_of(work, struct ivpu_device, irq_dct_work); bool enable; int ret; diff --git a/drivers/accel/ivpu/ivpu_pm.h b/drivers/accel/ivpu/ivpu_pm.h index b70efe6c36e4..89b264cc0e3e 100644 --- a/drivers/accel/ivpu/ivpu_pm.h +++ b/drivers/accel/ivpu/ivpu_pm.h @@ -45,6 +45,6 @@ void ivpu_stop_job_timeout_detection(struct ivpu_device *vdev); int ivpu_pm_dct_init(struct ivpu_device *vdev); int ivpu_pm_dct_enable(struct ivpu_device *vdev, u8 active_percent); int ivpu_pm_dct_disable(struct ivpu_device *vdev); -void ivpu_pm_dct_irq_thread_handler(struct ivpu_device *vdev); +void ivpu_pm_irq_dct_work_fn(struct work_struct *work); #endif /* __IVPU_PM_H__ */ From patchwork Tue Jan 7 17:32:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6298BE7719B for ; Tue, 7 Jan 2025 14:22:41 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D8C8510E702; Tue, 7 Jan 2025 14:22:40 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="nAxW4hw0"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0CCEA10E6FB for ; Tue, 7 Jan 2025 14:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259760; x=1767795760; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JYkD9Ku9R0T4O/HbMfreqFOqNIGIemnC2XIOZG9+FXU=; b=nAxW4hw0CZMN1spwKF8HZofCna/dkS1inKN/C2jVQbdFXO/NRpWtcvfA FWH5EyeTR/DZxQsUIGfpVd3N7XK1H4ceR7ZromZjiBv+4Gny56Khtey9d Yf5UBHrgVyhmt/7iPe966Ic4bFqKuv0ksoVn0BZXPdPBinfpL2Vonhsr1 YjfJqYWpCpYT9VxrYW8Opem5cq30s6vu/k5EOI2EOfoghbpmY8bbr5fzO o5JcHJOrYZyDxei5N7uVz1oWnUzWQg/WKP9b0vtyoNqlQjko0wYmboHlx 3DfnRLNaC1Ma2WSSTe4UP9TocqOxaOPUK9ylT0KyX49WIGkXdA7+ci6Fw Q==; X-CSE-ConnectionGUID: ke2UPW5uQuGZeLNHIS2IHw== X-CSE-MsgGUID: /vuTSxPGQQanx8uccIv/dg== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324439" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324439" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:39 -0800 X-CSE-ConnectionGUID: /v/iLgJaSXm4g8WSPEMeRQ== X-CSE-MsgGUID: HbdbzdThQkulV4isRHpOQA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635451" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:38 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 06/14] accel/ivpu: Dump only first MMU fault from single context Date: Tue, 7 Jan 2025 18:32:29 +0100 Message-ID: <20250107173238.381120-7-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Stop dumping consecutive faults from an already faulty context immediately, instead of waiting for the context abort thread handler (IRQ handler bottom half) to abort currently executing jobs. Remove 'R' (record events) bit from context descriptor of a faulty context to prevent future faults generation. This change speeds up the IRQ handler by eliminating the need to print the fault content repeatedly. Additionally, it prevents flooding dmesg with errors, which was occurring due to the delay in the bottom half of the handler stopping fault-generating jobs. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_mmu.c | 51 ++++++++++++++++++++++++--- drivers/accel/ivpu/ivpu_mmu_context.c | 13 ------- drivers/accel/ivpu/ivpu_mmu_context.h | 2 -- 3 files changed, 46 insertions(+), 20 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_mmu.c b/drivers/accel/ivpu/ivpu_mmu.c index 21f820dd0c65..5ee4df892b3e 100644 --- a/drivers/accel/ivpu/ivpu_mmu.c +++ b/drivers/accel/ivpu/ivpu_mmu.c @@ -870,23 +870,64 @@ static u32 *ivpu_mmu_get_event(struct ivpu_device *vdev) return evt; } +static int ivpu_mmu_disable_events(struct ivpu_device *vdev, u32 ssid) +{ + struct ivpu_mmu_info *mmu = vdev->mmu; + struct ivpu_mmu_cdtab *cdtab = &mmu->cdtab; + u64 *entry; + u64 val; + + if (ssid > IVPU_MMU_CDTAB_ENT_COUNT) + return -EINVAL; + + entry = cdtab->base + (ssid * IVPU_MMU_CDTAB_ENT_SIZE); + + val = READ_ONCE(entry[0]); + val &= ~IVPU_MMU_CD_0_R; + WRITE_ONCE(entry[0], val); + + if (!ivpu_is_force_snoop_enabled(vdev)) + clflush_cache_range(entry, IVPU_MMU_CDTAB_ENT_SIZE); + + ivpu_mmu_cmdq_write_cfgi_all(vdev); + + return 0; +} + void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) { + struct ivpu_file_priv *file_priv; + u32 last_ssid = -1; u32 *event; u32 ssid; ivpu_dbg(vdev, IRQ, "MMU event queue\n"); - while ((event = ivpu_mmu_get_event(vdev)) != NULL) { - ivpu_mmu_dump_event(vdev, event); - + while ((event = ivpu_mmu_get_event(vdev))) { ssid = FIELD_GET(IVPU_MMU_EVT_SSID_MASK, event[0]); + + if (ssid == last_ssid) + continue; + + xa_lock(&vdev->context_xa); + file_priv = xa_load(&vdev->context_xa, ssid); + if (file_priv) { + if (file_priv->has_mmu_faults) { + event = NULL; + } else { + ivpu_mmu_disable_events(vdev, ssid); + file_priv->has_mmu_faults = true; + } + } + xa_unlock(&vdev->context_xa); + + if (event) + ivpu_mmu_dump_event(vdev, event); + if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID) { ivpu_pm_trigger_recovery(vdev, "MMU event"); return; } - - ivpu_mmu_user_context_mark_invalid(vdev, ssid); REGV_WR32(IVPU_MMU_REG_EVTQ_CONS_SEC, vdev->mmu->evtq.cons); } diff --git a/drivers/accel/ivpu/ivpu_mmu_context.c b/drivers/accel/ivpu/ivpu_mmu_context.c index 891967a95bc3..d373443bbc83 100644 --- a/drivers/accel/ivpu/ivpu_mmu_context.c +++ b/drivers/accel/ivpu/ivpu_mmu_context.c @@ -631,16 +631,3 @@ void ivpu_mmu_reserved_context_fini(struct ivpu_device *vdev) ivpu_mmu_cd_clear(vdev, vdev->rctx.id); ivpu_mmu_context_fini(vdev, &vdev->rctx); } - -void ivpu_mmu_user_context_mark_invalid(struct ivpu_device *vdev, u32 ssid) -{ - struct ivpu_file_priv *file_priv; - - xa_lock(&vdev->context_xa); - - file_priv = xa_load(&vdev->context_xa, ssid); - if (file_priv) - file_priv->has_mmu_faults = true; - - xa_unlock(&vdev->context_xa); -} diff --git a/drivers/accel/ivpu/ivpu_mmu_context.h b/drivers/accel/ivpu/ivpu_mmu_context.h index 8042fc067062..f255310968cf 100644 --- a/drivers/accel/ivpu/ivpu_mmu_context.h +++ b/drivers/accel/ivpu/ivpu_mmu_context.h @@ -37,8 +37,6 @@ void ivpu_mmu_global_context_fini(struct ivpu_device *vdev); int ivpu_mmu_reserved_context_init(struct ivpu_device *vdev); void ivpu_mmu_reserved_context_fini(struct ivpu_device *vdev); -void ivpu_mmu_user_context_mark_invalid(struct ivpu_device *vdev, u32 ssid); - int ivpu_mmu_context_insert_node(struct ivpu_mmu_context *ctx, const struct ivpu_addr_range *range, u64 size, struct drm_mm_node *node); void ivpu_mmu_context_remove_node(struct ivpu_mmu_context *ctx, struct drm_mm_node *node); From patchwork Tue Jan 7 17:32:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 821DCE77197 for ; Tue, 7 Jan 2025 14:22:43 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 09C7210EA1A; Tue, 7 Jan 2025 14:22:43 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="C1UKCmP4"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id DE1E610E717 for ; Tue, 7 Jan 2025 14:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259762; x=1767795762; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kcX7O5hr8iid0nhQNJtcGyBqL4sfF0g8utxeZF6d4Gg=; b=C1UKCmP4fZ3FJiuHhnv/4A/0zyJZuNECyaWVbxj9VmehC6A2c5bCCrdh y/OxaJay2A84jrOhudb/sp8PEAKVpe4SE3fQHbwkOuPFeeRWfSTexiZ3V WXQAywKLqTJFO6rSqS3lnKnsbl7MBJHsA7Iceyz679YX/iqsCB1bwGa+V TScrRm/tGURn//q2RWHaPG+qcPcdiYO0XNI6PfKjew6oXcriW4u4/e8qe P87+6L57SKM4xpw4iHxoNW6MGFZuJKiTDq/ir/QXQfiZVsqy+E4EBq9B4 UZ4840aVevHXWMEfImOW5mJRldK+xOGAkYNNG0suDPCkEuQ7UcMUeCEr1 w==; X-CSE-ConnectionGUID: dZSQ2ZfcSfux9KM9ZaVc7A== X-CSE-MsgGUID: jpdBaWyIRDayI/r9M+TvNg== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324444" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324444" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:41 -0800 X-CSE-ConnectionGUID: q8OQh/2VR7ysAevTWZN+vw== X-CSE-MsgGUID: sFDdfmjvTvSPUfrdcwQ2ig== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635459" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:40 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 07/14] accel/ivpu: Move parts of MMU event IRQ handling to thread handler Date: Tue, 7 Jan 2025 18:32:30 +0100 Message-ID: <20250107173238.381120-8-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski To prevent looping infinitely in MMU event handler we stop generating new events by removing 'R' (record) bit from context descriptor, but to ensure this change has effect KMD has to perform configuration invalidation followed by sync command. Because of that move parts of the interrupt handler that can take longer to a thread not to block in interrupt handler for too long. This includes: * disabling event queue for the time KMD updates MMU event queue consumer to ensure proper synchronization between MMU and KMD * removal of 'R' (record) bit from context descriptor to ensure no more faults are recorded until that context is destroyed Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_job.c | 7 ++- drivers/accel/ivpu/ivpu_mmu.c | 93 +++++++++++++++++++++++------------ drivers/accel/ivpu/ivpu_mmu.h | 2 + 3 files changed, 69 insertions(+), 33 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index c678dcddb8d8..c55de9736d84 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -17,6 +17,7 @@ #include "ivpu_ipc.h" #include "ivpu_job.h" #include "ivpu_jsm_msg.h" +#include "ivpu_mmu.h" #include "ivpu_pm.h" #include "ivpu_trace.h" #include "vpu_boot_api.h" @@ -366,6 +367,7 @@ void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv) unsigned long cmdq_id; lockdep_assert_held(&file_priv->lock); + ivpu_dbg(vdev, JOB, "Context ID: %u abort\n", file_priv->ctx.id); xa_for_each(&file_priv->cmdq_xa, cmdq_id, cmdq) ivpu_cmdq_unregister(file_priv, cmdq); @@ -373,6 +375,9 @@ void ivpu_context_abort_locked(struct ivpu_file_priv *file_priv) if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_OS) ivpu_jsm_context_release(vdev, file_priv->ctx.id); + ivpu_mmu_disable_ssid_events(vdev, file_priv->ctx.id); + ivpu_mmu_discard_events(vdev); + file_priv->aborted = true; } @@ -939,8 +944,8 @@ void ivpu_context_abort_work_fn(struct work_struct *work) { struct ivpu_device *vdev = container_of(work, struct ivpu_device, context_abort_work); struct ivpu_file_priv *file_priv; - unsigned long ctx_id; struct ivpu_job *job; + unsigned long ctx_id; unsigned long id; mutex_lock(&vdev->context_list_lock); diff --git a/drivers/accel/ivpu/ivpu_mmu.c b/drivers/accel/ivpu/ivpu_mmu.c index 5ee4df892b3e..ae1dcd04051c 100644 --- a/drivers/accel/ivpu/ivpu_mmu.c +++ b/drivers/accel/ivpu/ivpu_mmu.c @@ -20,6 +20,12 @@ #define IVPU_MMU_REG_CR0 0x00200020u #define IVPU_MMU_REG_CR0ACK 0x00200024u #define IVPU_MMU_REG_CR0ACK_VAL_MASK GENMASK(31, 0) +#define IVPU_MMU_REG_CR0_ATSCHK_MASK BIT(4) +#define IVPU_MMU_REG_CR0_CMDQEN_MASK BIT(3) +#define IVPU_MMU_REG_CR0_EVTQEN_MASK BIT(2) +#define IVPU_MMU_REG_CR0_PRIQEN_MASK BIT(1) +#define IVPU_MMU_REG_CR0_SMMUEN_MASK BIT(0) + #define IVPU_MMU_REG_CR1 0x00200028u #define IVPU_MMU_REG_CR2 0x0020002cu #define IVPU_MMU_REG_IRQ_CTRL 0x00200050u @@ -141,12 +147,6 @@ #define IVPU_MMU_IRQ_EVTQ_EN BIT(2) #define IVPU_MMU_IRQ_GERROR_EN BIT(0) -#define IVPU_MMU_CR0_ATSCHK BIT(4) -#define IVPU_MMU_CR0_CMDQEN BIT(3) -#define IVPU_MMU_CR0_EVTQEN BIT(2) -#define IVPU_MMU_CR0_PRIQEN BIT(1) -#define IVPU_MMU_CR0_SMMUEN BIT(0) - #define IVPU_MMU_CR1_TABLE_SH GENMASK(11, 10) #define IVPU_MMU_CR1_TABLE_OC GENMASK(9, 8) #define IVPU_MMU_CR1_TABLE_IC GENMASK(7, 6) @@ -596,7 +596,7 @@ static int ivpu_mmu_reset(struct ivpu_device *vdev) REGV_WR32(IVPU_MMU_REG_CMDQ_PROD, 0); REGV_WR32(IVPU_MMU_REG_CMDQ_CONS, 0); - val = IVPU_MMU_CR0_CMDQEN; + val = REG_SET_FLD(IVPU_MMU_REG_CR0, CMDQEN, 0); ret = ivpu_mmu_reg_write_cr0(vdev, val); if (ret) return ret; @@ -617,12 +617,12 @@ static int ivpu_mmu_reset(struct ivpu_device *vdev) REGV_WR32(IVPU_MMU_REG_EVTQ_PROD_SEC, 0); REGV_WR32(IVPU_MMU_REG_EVTQ_CONS_SEC, 0); - val |= IVPU_MMU_CR0_EVTQEN; + val = REG_SET_FLD(IVPU_MMU_REG_CR0, EVTQEN, val); ret = ivpu_mmu_reg_write_cr0(vdev, val); if (ret) return ret; - val |= IVPU_MMU_CR0_ATSCHK; + val = REG_SET_FLD(IVPU_MMU_REG_CR0, ATSCHK, val); ret = ivpu_mmu_reg_write_cr0(vdev, val); if (ret) return ret; @@ -631,7 +631,7 @@ static int ivpu_mmu_reset(struct ivpu_device *vdev) if (ret) return ret; - val |= IVPU_MMU_CR0_SMMUEN; + val = REG_SET_FLD(IVPU_MMU_REG_CR0, SMMUEN, val); return ivpu_mmu_reg_write_cr0(vdev, val); } @@ -870,7 +870,47 @@ static u32 *ivpu_mmu_get_event(struct ivpu_device *vdev) return evt; } -static int ivpu_mmu_disable_events(struct ivpu_device *vdev, u32 ssid) +static int ivpu_mmu_evtq_set(struct ivpu_device *vdev, bool enable) +{ + u32 val = REGV_RD32(IVPU_MMU_REG_CR0); + + if (enable) + val = REG_SET_FLD(IVPU_MMU_REG_CR0, EVTQEN, val); + else + val = REG_CLR_FLD(IVPU_MMU_REG_CR0, EVTQEN, val); + REGV_WR32(IVPU_MMU_REG_CR0, val); + + return REGV_POLL_FLD(IVPU_MMU_REG_CR0ACK, VAL, val, IVPU_MMU_REG_TIMEOUT_US); +} + +static int ivpu_mmu_evtq_enable(struct ivpu_device *vdev) +{ + return ivpu_mmu_evtq_set(vdev, true); +} + +static int ivpu_mmu_evtq_disable(struct ivpu_device *vdev) +{ + return ivpu_mmu_evtq_set(vdev, false); +} + +void ivpu_mmu_discard_events(struct ivpu_device *vdev) +{ + /* + * Disable event queue (stop MMU from updating the producer) + * to allow synchronization of consumer and producer indexes + */ + ivpu_mmu_evtq_disable(vdev); + + vdev->mmu->evtq.cons = REGV_RD32(IVPU_MMU_REG_EVTQ_PROD_SEC); + REGV_WR32(IVPU_MMU_REG_EVTQ_CONS_SEC, vdev->mmu->evtq.cons); + vdev->mmu->evtq.prod = REGV_RD32(IVPU_MMU_REG_EVTQ_PROD_SEC); + + ivpu_mmu_evtq_enable(vdev); + + drm_WARN_ON_ONCE(&vdev->drm, vdev->mmu->evtq.cons != vdev->mmu->evtq.prod); +} + +int ivpu_mmu_disable_ssid_events(struct ivpu_device *vdev, u32 ssid) { struct ivpu_mmu_info *mmu = vdev->mmu; struct ivpu_mmu_cdtab *cdtab = &mmu->cdtab; @@ -890,6 +930,7 @@ static int ivpu_mmu_disable_events(struct ivpu_device *vdev, u32 ssid) clflush_cache_range(entry, IVPU_MMU_CDTAB_ENT_SIZE); ivpu_mmu_cmdq_write_cfgi_all(vdev); + ivpu_mmu_cmdq_sync(vdev); return 0; } @@ -897,38 +938,26 @@ static int ivpu_mmu_disable_events(struct ivpu_device *vdev, u32 ssid) void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) { struct ivpu_file_priv *file_priv; - u32 last_ssid = -1; u32 *event; u32 ssid; ivpu_dbg(vdev, IRQ, "MMU event queue\n"); while ((event = ivpu_mmu_get_event(vdev))) { - ssid = FIELD_GET(IVPU_MMU_EVT_SSID_MASK, event[0]); - - if (ssid == last_ssid) - continue; + ssid = FIELD_GET(IVPU_MMU_EVT_SSID_MASK, *event); + if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID) { + ivpu_mmu_dump_event(vdev, event); + ivpu_pm_trigger_recovery(vdev, "MMU event"); + return; + } - xa_lock(&vdev->context_xa); file_priv = xa_load(&vdev->context_xa, ssid); if (file_priv) { - if (file_priv->has_mmu_faults) { - event = NULL; - } else { - ivpu_mmu_disable_events(vdev, ssid); - file_priv->has_mmu_faults = true; + if (!READ_ONCE(file_priv->has_mmu_faults)) { + ivpu_mmu_dump_event(vdev, event); + WRITE_ONCE(file_priv->has_mmu_faults, true); } } - xa_unlock(&vdev->context_xa); - - if (event) - ivpu_mmu_dump_event(vdev, event); - - if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID) { - ivpu_pm_trigger_recovery(vdev, "MMU event"); - return; - } - REGV_WR32(IVPU_MMU_REG_EVTQ_CONS_SEC, vdev->mmu->evtq.cons); } queue_work(system_wq, &vdev->context_abort_work); diff --git a/drivers/accel/ivpu/ivpu_mmu.h b/drivers/accel/ivpu/ivpu_mmu.h index 7afea9cd8731..1ce7529746ad 100644 --- a/drivers/accel/ivpu/ivpu_mmu.h +++ b/drivers/accel/ivpu/ivpu_mmu.h @@ -47,5 +47,7 @@ int ivpu_mmu_invalidate_tlb(struct ivpu_device *vdev, u16 ssid); void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev); void ivpu_mmu_irq_gerr_handler(struct ivpu_device *vdev); void ivpu_mmu_evtq_dump(struct ivpu_device *vdev); +void ivpu_mmu_discard_events(struct ivpu_device *vdev); +int ivpu_mmu_disable_ssid_events(struct ivpu_device *vdev, u32 ssid); #endif /* __IVPU_MMU_H__ */ From patchwork Tue Jan 7 17:32:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 76D3EE77198 for ; Tue, 7 Jan 2025 14:22:45 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E3F5410EABE; Tue, 7 Jan 2025 14:22:44 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="d5Pa6HpR"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id C0F8D10EAC7 for ; Tue, 7 Jan 2025 14:22:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259763; x=1767795763; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SkMdvjIWu78uAx/5XaEfIot2OY4QlhppHKKuETt5wvg=; b=d5Pa6HpR5r2eUxO8F+5UW0vPk+W3NfAmcXEkhOfPB3tDg71uays+lX9n oLQ9EB7tCAkGXNXEb9n0xNyabflVUFkBmbltWPhEPZsTytn04Fry6Dv0E 2QNGvHc7xXUvXrENHs6pwFBI8fvcayOb9a11mIEoVY3SRwEbLPTSw3Sou VtUTLa46UNFbrTDlB7h0KTtpeaz3yYMPhYPI7fk+pCrlvK33GZgIhAu8i oTAxtq80aTkpUM+Bar5enJ3bxtzjlgPBTYWZ+Jh9n2hlfJK3vUnj0q/dT GXwlYaTVGLeX0doA3C4EUa1zYVvnz0UXJMrqnwD/3qLCzvOySKYUNM26H Q==; X-CSE-ConnectionGUID: 73g4RJpAR86ox3miF+aOBw== X-CSE-MsgGUID: xfS7FTncSQ2kvFVxgwjytw== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324448" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324448" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:43 -0800 X-CSE-ConnectionGUID: 2HvIMn0iRlCptsB9Q4rs5g== X-CSE-MsgGUID: aV/HXyovSDq5PZPiUYerWw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635465" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:42 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 08/14] accel/ivpu: Fix missing MMU events from reserved SSID Date: Tue, 7 Jan 2025 18:32:31 +0100 Message-ID: <20250107173238.381120-9-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Generate recovery when fault from reserved context is detected. Add Abort (A) bit to reserved (1) SSID to ensure NPU also receives a fault. There is no way to create a file_priv with reserved SSID but it is still possible to receive MMU faults from that SSID as it is a default NPU HW setting. Such situation will occur if FW freed context related resources but still performed access to DRAM. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_mmu.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_mmu.c b/drivers/accel/ivpu/ivpu_mmu.c index ae1dcd04051c..b80bdded9fd7 100644 --- a/drivers/accel/ivpu/ivpu_mmu.c +++ b/drivers/accel/ivpu/ivpu_mmu.c @@ -725,8 +725,8 @@ static int ivpu_mmu_cdtab_entry_set(struct ivpu_device *vdev, u32 ssid, u64 cd_d cd[2] = 0; cd[3] = 0x0000000000007444; - /* For global context generate memory fault on VPU */ - if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID) + /* For global and reserved contexts generate memory fault on VPU */ + if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID || ssid == IVPU_RESERVED_CONTEXT_MMU_SSID) cd[0] |= IVPU_MMU_CD_0_A; if (valid) @@ -945,7 +945,8 @@ void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) while ((event = ivpu_mmu_get_event(vdev))) { ssid = FIELD_GET(IVPU_MMU_EVT_SSID_MASK, *event); - if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID) { + if (ssid == IVPU_GLOBAL_CONTEXT_MMU_SSID || + ssid == IVPU_RESERVED_CONTEXT_MMU_SSID) { ivpu_mmu_dump_event(vdev, event); ivpu_pm_trigger_recovery(vdev, "MMU event"); return; From patchwork Tue Jan 7 17:32:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D4951E77197 for ; Tue, 7 Jan 2025 14:22:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5B89C10E6FB; Tue, 7 Jan 2025 14:22:47 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="BINaFZjQ"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 96BAE10E6FB for ; Tue, 7 Jan 2025 14:22:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259765; x=1767795765; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rWyldU0O6dmHH3v5FxIb7hS4ISh5mElv37BxXipdo2g=; b=BINaFZjQuWl4XfHd3btxh6YE072cHuxsKed9x6tGb6TG0Zf8BBxKEzNT MLcHEq0ITrmudhXtJgtCOesc9oFg4j9432LLxFWhG1fHJXW1lKKmz7+Qd HfSVR5KOu1wfGGC3wsi9HL83C6XxYGvAZE5+X7izwelVavWROeg2HJNeM i1uRyr4BsbcAk1MQRjWFZwxP1i7WQRxJaaWK08v1J1wTzdArb9x1rW7iC aU47zps4bfJA3Av7/H2ZjC4t14ye8Wz/jq/i5R2zLYt1Fmum/98lJCK38 JlQwqfSI0Mq9I3XVFrI0RlkL4xshlDC8j5ENScxjCWaxP70CQKEIvGhN5 Q==; X-CSE-ConnectionGUID: /4B6s7/7Qtajuie6f1LDfg== X-CSE-MsgGUID: AzS1n+CWR5ac75rf1XXHVw== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324455" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324455" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:45 -0800 X-CSE-ConnectionGUID: ObjW/04nSEOKT2otOPME4Q== X-CSE-MsgGUID: qMuBj2eeThubQiQfu5B96Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635473" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:44 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 09/14] accel/ivpu: Set command queue management capability based on HWS Date: Tue, 7 Jan 2025 18:32:32 +0100 Message-ID: <20250107173238.381120-10-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Control explicit command queue management capability bit based on scheduling mode. Capability will be available only when hardware scheduling mode is set. There is no point of allowing user space to create and destroy command queues with OS schedling mode because FW does not support all required functionalities for correct command queue management with OS scheduling. Return -ENODEV from command queue create/destroy/submit IOCTLs. Remove is_valid field from struct ivpu_job_cmdq Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_drv.c | 19 +++++++------------ drivers/accel/ivpu/ivpu_drv.h | 1 + drivers/accel/ivpu/ivpu_job.c | 24 +++++++++++------------- drivers/accel/ivpu/ivpu_job.h | 1 - 4 files changed, 19 insertions(+), 26 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_drv.c b/drivers/accel/ivpu/ivpu_drv.c index 9b0d99873fb3..6a80d626d609 100644 --- a/drivers/accel/ivpu/ivpu_drv.c +++ b/drivers/accel/ivpu/ivpu_drv.c @@ -127,23 +127,18 @@ void ivpu_file_priv_put(struct ivpu_file_priv **link) kref_put(&file_priv->ref, file_priv_release); } -static int ivpu_get_capabilities(struct ivpu_device *vdev, struct drm_ivpu_param *args) +bool ivpu_is_capable(struct ivpu_device *vdev, u32 capability) { - switch (args->index) { + switch (capability) { case DRM_IVPU_CAP_METRIC_STREAMER: - args->value = 1; - break; + return true; case DRM_IVPU_CAP_DMA_MEMORY_RANGE: - args->value = 1; - break; + return true; case DRM_IVPU_CAP_MANAGE_CMDQ: - args->value = 1; - break; + return vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW; default: - return -EINVAL; + return false; } - - return 0; } static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_file *file) @@ -203,7 +198,7 @@ static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_f args->value = vdev->hw->sku; break; case DRM_IVPU_PARAM_CAPABILITIES: - ret = ivpu_get_capabilities(vdev, args); + args->value = ivpu_is_capable(vdev, args->index); break; default: ret = -EINVAL; diff --git a/drivers/accel/ivpu/ivpu_drv.h b/drivers/accel/ivpu/ivpu_drv.h index b57d878f2fcd..d53902b34070 100644 --- a/drivers/accel/ivpu/ivpu_drv.h +++ b/drivers/accel/ivpu/ivpu_drv.h @@ -213,6 +213,7 @@ void ivpu_file_priv_put(struct ivpu_file_priv **link); int ivpu_boot(struct ivpu_device *vdev); int ivpu_shutdown(struct ivpu_device *vdev); void ivpu_prepare_for_reset(struct ivpu_device *vdev); +bool ivpu_is_capable(struct ivpu_device *vdev, u32 capability); static inline u8 ivpu_revision(struct ivpu_device *vdev) { diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index c55de9736d84..37ea92eb4b25 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -123,7 +123,6 @@ static struct ivpu_cmdq *ivpu_cmdq_create(struct ivpu_file_priv *file_priv, u8 p cmdq->priority = priority; cmdq->is_legacy = is_legacy; - cmdq->is_valid = true; ret = xa_alloc_cyclic(&file_priv->cmdq_xa, &cmdq->id, cmdq, file_priv->cmdq_limit, &file_priv->cmdq_id_next, GFP_KERNEL); @@ -307,7 +306,7 @@ static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u32 lockdep_assert_held(&file_priv->lock); cmdq = xa_load(&file_priv->cmdq_xa, cmdq_id); - if (!cmdq || !cmdq->is_valid) { + if (!cmdq) { ivpu_warn_ratelimited(vdev, "Failed to find command queue with ID: %u\n", cmdq_id); return NULL; } @@ -832,6 +831,9 @@ int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file * struct ivpu_file_priv *file_priv = file->driver_priv; struct drm_ivpu_cmdq_submit *args = data; + if (!ivpu_is_capable(file_priv->vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) + return -ENODEV; + if (args->cmdq_id < IVPU_CMDQ_MIN_ID || args->cmdq_id > IVPU_CMDQ_MAX_ID) return -EINVAL; @@ -857,6 +859,9 @@ int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file * struct drm_ivpu_cmdq_create *args = data; struct ivpu_cmdq *cmdq; + if (!ivpu_is_capable(file_priv->vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) + return -ENODEV; + if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) return -EINVAL; @@ -880,24 +885,17 @@ int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file u32 cmdq_id; int ret = 0; + if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) + return -ENODEV; + mutex_lock(&file_priv->lock); cmdq = xa_load(&file_priv->cmdq_xa, args->cmdq_id); - if (!cmdq || !cmdq->is_valid || cmdq->is_legacy) { + if (!cmdq || cmdq->is_legacy) { ret = -ENOENT; goto unlock; } - /* - * There is no way to stop executing jobs per command queue - * in OS scheduling mode, mark command queue as invalid instead - * and it will be freed together with context release. - */ - if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_OS) { - cmdq->is_valid = false; - goto unlock; - } - cmdq_id = cmdq->id; ivpu_cmdq_destroy(file_priv, cmdq); ivpu_cmdq_abort_all_jobs(vdev, file_priv->ctx.id, cmdq_id); diff --git a/drivers/accel/ivpu/ivpu_job.h b/drivers/accel/ivpu/ivpu_job.h index ff77ee1fcee2..2e301c2eea7b 100644 --- a/drivers/accel/ivpu/ivpu_job.h +++ b/drivers/accel/ivpu/ivpu_job.h @@ -31,7 +31,6 @@ struct ivpu_cmdq { u32 id; u32 db_id; u8 priority; - bool is_valid; bool is_legacy; }; From patchwork Tue Jan 7 17:32:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8F96E7719A for ; Tue, 7 Jan 2025 14:22:49 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 51DC110EAC7; Tue, 7 Jan 2025 14:22:49 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="iFf8maRA"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 747CA10E708 for ; Tue, 7 Jan 2025 14:22:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259767; x=1767795767; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iyR050BA/SGkKlwz6I9825V1/9w0d/jviKByeI5h/Co=; b=iFf8maRA53RMifBOmJyb5jaRU+4MwGuNDRWgGvGhISWd+flgHQFTnQED wXVLoDz04gJGSA0Ssvwl1atkGPbr5ej53Va+ME0VUk5zLI9J4cNyTZSY/ z2jifPPaQtM+h9vCY454eHcyGIIHDDzcEuGmZFFkwrsU0DAlWf1l2jN8k dMqG/DwrnQgit5OiF/v0x+8wVT9bmfqZu3HUQmqNqpnUQHubnwNhZfhx5 kPEHMWBoez1bGGBF+ubG1Yejn34guBWDOEAwb81ItB1sxAxgeaG9zgLBI DEvY3pqudN5EOJHF711GsihLvbj5yo7tn3ABpCXcpAqMFVecPcnYXSk0m w==; X-CSE-ConnectionGUID: o2fxbFcwTzKLE5VrTgT7+g== X-CSE-MsgGUID: 93GDaB6BTse8NJ815VKTDg== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324462" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324462" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:47 -0800 X-CSE-ConnectionGUID: l3yAfN9aR5KO6+29SehQkw== X-CSE-MsgGUID: ecNEC/zYRwWVU022vhYZ9A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635483" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:46 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 10/14] accel/ivpu: Fix locking order in ivpu_cmdq_destroy_ioctl Date: Tue, 7 Jan 2025 18:32:33 +0100 Message-ID: <20250107173238.381120-11-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Fix deadlock caused by inversed locking order in ivpu_job_submit() and ivpu_cmdq_destroy_ioctl(). Both functions operate locking file_priv->lock and submitted_jobs_lock. Unlock file_priv->lock in ivpu_cmdq_destroy_ioctl() before calling ivpu_cmdq_abort_all_jobs() function which locks submitted_jobs_lock. That way locking order is maintained: 1) global submitted_jobs_lock first 2) per context file_priv->lock second Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_job.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index 37ea92eb4b25..c694822a14bf 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -883,7 +883,7 @@ int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file struct drm_ivpu_cmdq_destroy *args = data; struct ivpu_cmdq *cmdq; u32 cmdq_id; - int ret = 0; + int ret; if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) return -ENODEV; @@ -893,13 +893,16 @@ int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file cmdq = xa_load(&file_priv->cmdq_xa, args->cmdq_id); if (!cmdq || cmdq->is_legacy) { ret = -ENOENT; - goto unlock; + goto err_unlock; } cmdq_id = cmdq->id; ivpu_cmdq_destroy(file_priv, cmdq); + mutex_unlock(&file_priv->lock); ivpu_cmdq_abort_all_jobs(vdev, file_priv->ctx.id, cmdq_id); -unlock: + return 0; + +err_unlock: mutex_unlock(&file_priv->lock); return ret; } From patchwork Tue Jan 7 17:32:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9CB50E77198 for ; Tue, 7 Jan 2025 14:22:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EF1FC10E708; Tue, 7 Jan 2025 14:22:50 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="ZxECE9wx"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5501B10EAD1 for ; Tue, 7 Jan 2025 14:22:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259769; x=1767795769; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fDeSjr2N0ZgZwEYt5benJYoBhxMK1Fls1tYNKedyiUA=; b=ZxECE9wxpeH80cdKh52HYw29+E1jvl1w2R5+1whl8hgkrt5gf0Z9xJZN mZXgftoW5kIKu4hyC1g7QFL6YehylvoIvggyHJVtPkKqgH/Bxq6xfjVDV quA46qcG7WiTmJO/rIYoTfo9dccMQhIZi36eS1pQj6B1x23kWIYsmB5hz OReLCdLUI8VW+TpxfKJmwL31sDHl/8feEi/4KTgxEGSE2C4odnt0yKhXD R3khV6qkIFGw+CKw85SCx2ibyedZRAqnvAxXcLRZqae7HiwWZ+opTXGmA g5vWQp4EtZB3zpNrqoiM3zl0O+Y7DeyqpZXpSwMcGt6lTqYemJbcyU3Tq g==; X-CSE-ConnectionGUID: b/ifGnimQ+e/UaK3fqH96w== X-CSE-MsgGUID: 7Nb3a0skS7KjVH+y/rcNew== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324469" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324469" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:49 -0800 X-CSE-ConnectionGUID: 38Mjp+IyTQi1L2yIXTefqg== X-CSE-MsgGUID: HLfMTR8eRnqMzwJO4HnhKQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635497" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:48 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 11/14] accel/ivpu: Fix locking order in ivpu_job_submit Date: Tue, 7 Jan 2025 18:32:34 +0100 Message-ID: <20250107173238.381120-12-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Fix deadlock in job submission and abort handling. When a thread aborts currently executing jobs due to a fault, it first locks the global lock protecting submitted_jobs (#1). After the last job is destroyed, it proceeds to release the related context and locks file_priv (#2). Meanwhile, in the job submission thread, the file_priv lock (#2) is taken first, and then the submitted_jobs lock (#1) is obtained when a job is added to the submitted jobs list. CPU0 CPU1 ---- ---- (for example due to a fault) (jobs submissions keep coming) lock(&vdev->submitted_jobs_lock) #1 ivpu_jobs_abort_all() job_destroy() lock(&file_priv->lock) #2 lock(&vdev->submitted_jobs_lock) #1 file_priv_release() lock(&vdev->context_list_lock) lock(&file_priv->lock) #2 This order of locking causes a deadlock. To resolve this issue, change the order of locking in ivpu_job_submit(). Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_job.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index c694822a14bf..c93ea37062d7 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -597,6 +597,7 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) if (ret < 0) return ret; + mutex_lock(&vdev->submitted_jobs_lock); mutex_lock(&file_priv->lock); if (cmdq_id == 0) @@ -606,19 +607,17 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) if (!cmdq) { ivpu_warn_ratelimited(vdev, "Failed to get job queue, ctx %d\n", file_priv->ctx.id); ret = -EINVAL; - goto err_unlock_file_priv; + goto err_unlock; } ret = ivpu_cmdq_register(file_priv, cmdq); if (ret) { ivpu_err(vdev, "Failed to register command queue: %d\n", ret); - goto err_unlock_file_priv; + goto err_unlock; } job->cmdq_id = cmdq->id; - mutex_lock(&vdev->submitted_jobs_lock); - is_first_job = xa_empty(&vdev->submitted_jobs_xa); ret = xa_alloc_cyclic(&vdev->submitted_jobs_xa, &job->job_id, job, file_priv->job_limit, &file_priv->job_id_next, GFP_KERNEL); @@ -626,7 +625,7 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) ivpu_dbg(vdev, JOB, "Too many active jobs in ctx %d\n", file_priv->ctx.id); ret = -EBUSY; - goto err_unlock_submitted_jobs; + goto err_unlock; } ret = ivpu_cmdq_push_job(cmdq, job); @@ -649,22 +648,20 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) job->job_id, file_priv->ctx.id, job->engine_idx, cmdq->priority, job->cmd_buf_vpu_addr, cmdq->jobq->header.tail); - mutex_unlock(&vdev->submitted_jobs_lock); mutex_unlock(&file_priv->lock); if (unlikely(ivpu_test_mode & IVPU_TEST_MODE_NULL_HW)) { - mutex_lock(&vdev->submitted_jobs_lock); ivpu_job_signal_and_destroy(vdev, job->job_id, VPU_JSM_STATUS_SUCCESS); - mutex_unlock(&vdev->submitted_jobs_lock); } + mutex_unlock(&vdev->submitted_jobs_lock); + return 0; err_erase_xa: xa_erase(&vdev->submitted_jobs_xa, job->job_id); -err_unlock_submitted_jobs: +err_unlock: mutex_unlock(&vdev->submitted_jobs_lock); -err_unlock_file_priv: mutex_unlock(&file_priv->lock); ivpu_rpm_put(vdev); return ret; From patchwork Tue Jan 7 17:32:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CC472E77199 for ; Tue, 7 Jan 2025 14:22:52 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 46A6710EAD5; Tue, 7 Jan 2025 14:22:52 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="QL9WAwPt"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2FF7A10EAD2 for ; Tue, 7 Jan 2025 14:22:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259771; x=1767795771; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C26uqehIV44Ey7h6fgbfatipHKXpWPX0QHGSujbPVGQ=; b=QL9WAwPtrTaVs9debZW/S9EG/DtbMYdDlDBbt/vklOXJVoz7bCclWcUb /uw4Ye72vUTeZtxfDZS1bZIxM2xaEp1xSfgLPgDC2JdOKJRZZAW05Fp/P rkFBWF9HXentDxPUUNfZNJTVRYNrdfp50ZXwr/Pi10y/XWNDKIZmuZMXn ofjsE17WbHbDm98iIRkXcrYPxvfEdWoj5MhdzugLHCCmsdDcwsIutQ11z jkCfXbDBjIban0Ka0mD/k21h6VgQGqCXNtWd3EnJDAMRpQTCTyiVwrPkj HVe7d/4WHTW1w4iGzBzkFjW71zm0tDX8gMmG5eQQMQDhNGnwwn2QCA58c Q==; X-CSE-ConnectionGUID: nC60pkvHSPmoK6FF7fPpUw== X-CSE-MsgGUID: +WwZzturSm+GoZb9LnYi3g== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324473" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324473" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:51 -0800 X-CSE-ConnectionGUID: GACU82ruSrmXlAbd2T0ylA== X-CSE-MsgGUID: m19jbJsmR0yXxIufBTwClA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635510" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:49 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 12/14] accel/ivpu: Add handling of VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW Date: Tue, 7 Jan 2025 18:32:35 +0100 Message-ID: <20250107173238.381120-13-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Mark as invalid context of a job that returned HW context violation error and queue work that aborts jobs from faulty context. Add engine reset to the context abort thread handler to not only abort currently executing jobs but also to ensure NPU invalid state recovery. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_job.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/accel/ivpu/ivpu_job.c b/drivers/accel/ivpu/ivpu_job.c index c93ea37062d7..3c162ac41a1d 100644 --- a/drivers/accel/ivpu/ivpu_job.c +++ b/drivers/accel/ivpu/ivpu_job.c @@ -533,6 +533,26 @@ static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 lockdep_assert_held(&vdev->submitted_jobs_lock); + job = xa_load(&vdev->submitted_jobs_xa, job_id); + if (!job) + return -ENOENT; + + if (job_status == VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW) { + guard(mutex)(&job->file_priv->lock); + + if (job->file_priv->has_mmu_faults) + return 0; + + /* + * Mark context as faulty and defer destruction of the job to jobs abort thread + * handler to synchronize between both faults and jobs returning context violation + * status and ensure both are handled in the same way + */ + job->file_priv->has_mmu_faults = true; + queue_work(system_wq, &vdev->context_abort_work); + return 0; + } + job = ivpu_job_remove_from_submitted_jobs(vdev, job_id); if (!job) return -ENOENT; @@ -946,6 +966,9 @@ void ivpu_context_abort_work_fn(struct work_struct *work) unsigned long ctx_id; unsigned long id; + if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) + ivpu_jsm_reset_engine(vdev, 0); + mutex_lock(&vdev->context_list_lock); xa_for_each(&vdev->context_xa, ctx_id, file_priv) { if (!file_priv->has_mmu_faults || file_priv->aborted) @@ -959,6 +982,8 @@ void ivpu_context_abort_work_fn(struct work_struct *work) if (vdev->fw->sched_mode != VPU_SCHEDULING_MODE_HW) return; + + ivpu_jsm_hws_resume_engine(vdev, 0); /* * In hardware scheduling mode NPU already has stopped processing jobs * and won't send us any further notifications, thus we have to free job related resources From patchwork Tue Jan 7 17:32:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0CFDE7719A for ; Tue, 7 Jan 2025 14:22:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 6BA1410EAD1; Tue, 7 Jan 2025 14:22:54 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="A9IMGokp"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1797010EAD2 for ; Tue, 7 Jan 2025 14:22:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259773; x=1767795773; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RMqn4I8rzh/lt0yHX+cPTC3I+Q60fmJ8UFKQUazFO8g=; b=A9IMGokp1N7fMbcHQ452wkx06eV0lEaBFmsWqiJLLH8909t7/Zxz1/cG yb2BQ/+LLgu62LILI2ImICWM9ppJi19NvFygDRmna0ChsUoSiwc/CUilI p6pSpfvhcFtsVbS7oI8HQSY1Lzmc6nwrj3W0dziBFL6+bO+Eca0P34HuT HsyTNh6YHfm3y64xogbWuX3q16uIWx1VGRYZB9A37I6hVIH/jzHs8yOvw lWcdHl3utTXImaYJ8qikvsLPWKilpiLUArH+AsoUG5eUTCUoRIscFtovi 3qK/OXHvov+bu6ezm8GBpkKi42pu4GaZ18AOBFnyvLrDjLOldeDipsPy0 g==; X-CSE-ConnectionGUID: 8tQtP437Q4u6P21UFdkckA== X-CSE-MsgGUID: hhQKjUMdRm+N5nynWIv7XQ== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324479" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324479" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:52 -0800 X-CSE-ConnectionGUID: cklt1XOgSeahrmapi+1Yzw== X-CSE-MsgGUID: chl8s7EMRz2V2QKazmjNSQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635517" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:51 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Karol Wachowski , Maciej Falkowski Subject: [PATCH 13/14] accel/ivpu: Add platform detection for presilicon Date: Tue, 7 Jan 2025 18:32:36 +0100 Message-ID: <20250107173238.381120-14-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Karol Wachowski Use highest buttress VPU_STATUS register bits(15:13) that encode platform type as follows: 0 - Silicon 2 - Simics 3 - FPGA 4 - Hybrid SLE Remove old DMI based method. Signed-off-by: Karol Wachowski Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_drv.h | 4 ++- drivers/accel/ivpu/ivpu_hw.c | 41 ++++++++++------------- drivers/accel/ivpu/ivpu_hw_btrs.c | 7 ++++ drivers/accel/ivpu/ivpu_hw_btrs.h | 1 + drivers/accel/ivpu/ivpu_hw_btrs_lnl_reg.h | 1 + 5 files changed, 29 insertions(+), 25 deletions(-) diff --git a/drivers/accel/ivpu/ivpu_drv.h b/drivers/accel/ivpu/ivpu_drv.h index d53902b34070..ca21102ca366 100644 --- a/drivers/accel/ivpu/ivpu_drv.h +++ b/drivers/accel/ivpu/ivpu_drv.h @@ -58,6 +58,7 @@ #define IVPU_PLATFORM_SILICON 0 #define IVPU_PLATFORM_SIMICS 2 #define IVPU_PLATFORM_FPGA 3 +#define IVPU_PLATFORM_HSLE 4 #define IVPU_PLATFORM_INVALID 8 #define IVPU_SCHED_MODE_AUTO -1 @@ -288,7 +289,8 @@ static inline bool ivpu_is_simics(struct ivpu_device *vdev) static inline bool ivpu_is_fpga(struct ivpu_device *vdev) { - return ivpu_get_platform(vdev) == IVPU_PLATFORM_FPGA; + return ivpu_get_platform(vdev) == IVPU_PLATFORM_FPGA || + ivpu_get_platform(vdev) == IVPU_PLATFORM_HSLE; } static inline bool ivpu_is_force_snoop_enabled(struct ivpu_device *vdev) diff --git a/drivers/accel/ivpu/ivpu_hw.c b/drivers/accel/ivpu/ivpu_hw.c index 1b691375ee4d..e332f19ab51d 100644 --- a/drivers/accel/ivpu/ivpu_hw.c +++ b/drivers/accel/ivpu/ivpu_hw.c @@ -19,38 +19,31 @@ static char *platform_to_str(u32 platform) return "SIMICS"; case IVPU_PLATFORM_FPGA: return "FPGA"; + case IVPU_PLATFORM_HSLE: + return "HSLE"; default: return "Invalid platform"; } } -static const struct dmi_system_id dmi_platform_simulation[] = { - { - .ident = "Intel Simics", - .matches = { - DMI_MATCH(DMI_BOARD_NAME, "lnlrvp"), - DMI_MATCH(DMI_BOARD_VERSION, "1.0"), - DMI_MATCH(DMI_BOARD_SERIAL, "123456789"), - }, - }, - { - .ident = "Intel Simics", - .matches = { - DMI_MATCH(DMI_BOARD_NAME, "Simics"), - }, - }, - { } -}; - static void platform_init(struct ivpu_device *vdev) { - if (dmi_check_system(dmi_platform_simulation)) - vdev->platform = IVPU_PLATFORM_SIMICS; - else - vdev->platform = IVPU_PLATFORM_SILICON; + int platform = ivpu_hw_btrs_platform_read(vdev); + + ivpu_dbg(vdev, MISC, "Platform type: %s (%d)\n", platform_to_str(platform), platform); + + switch (platform) { + case IVPU_PLATFORM_SILICON: + case IVPU_PLATFORM_SIMICS: + case IVPU_PLATFORM_FPGA: + case IVPU_PLATFORM_HSLE: + vdev->platform = platform; + break; - ivpu_dbg(vdev, MISC, "Platform type: %s (%d)\n", - platform_to_str(vdev->platform), vdev->platform); + default: + ivpu_err(vdev, "Invalid platform type: %d\n", platform); + break; + } } static void wa_init(struct ivpu_device *vdev) diff --git a/drivers/accel/ivpu/ivpu_hw_btrs.c b/drivers/accel/ivpu/ivpu_hw_btrs.c index 3753b00ed2d6..56c56012b980 100644 --- a/drivers/accel/ivpu/ivpu_hw_btrs.c +++ b/drivers/accel/ivpu/ivpu_hw_btrs.c @@ -887,3 +887,10 @@ void ivpu_hw_btrs_diagnose_failure(struct ivpu_device *vdev) else return diagnose_failure_lnl(vdev); } + +int ivpu_hw_btrs_platform_read(struct ivpu_device *vdev) +{ + u32 reg = REGB_RD32(VPU_HW_BTRS_LNL_VPU_STATUS); + + return REG_GET_FLD(VPU_HW_BTRS_LNL_VPU_STATUS, PLATFORM, reg); +} diff --git a/drivers/accel/ivpu/ivpu_hw_btrs.h b/drivers/accel/ivpu/ivpu_hw_btrs.h index 04f14f50fed6..1fd71b4d4ab0 100644 --- a/drivers/accel/ivpu/ivpu_hw_btrs.h +++ b/drivers/accel/ivpu/ivpu_hw_btrs.h @@ -46,5 +46,6 @@ void ivpu_hw_btrs_global_int_disable(struct ivpu_device *vdev); void ivpu_hw_btrs_irq_enable(struct ivpu_device *vdev); void ivpu_hw_btrs_irq_disable(struct ivpu_device *vdev); void ivpu_hw_btrs_diagnose_failure(struct ivpu_device *vdev); +int ivpu_hw_btrs_platform_read(struct ivpu_device *vdev); #endif /* __IVPU_HW_BTRS_H__ */ diff --git a/drivers/accel/ivpu/ivpu_hw_btrs_lnl_reg.h b/drivers/accel/ivpu/ivpu_hw_btrs_lnl_reg.h index fc51f3098f97..fff2ef2cada6 100644 --- a/drivers/accel/ivpu/ivpu_hw_btrs_lnl_reg.h +++ b/drivers/accel/ivpu/ivpu_hw_btrs_lnl_reg.h @@ -86,6 +86,7 @@ #define VPU_HW_BTRS_LNL_VPU_STATUS_POWER_RESOURCE_OWN_ACK_MASK BIT_MASK(7) #define VPU_HW_BTRS_LNL_VPU_STATUS_PERF_CLK_MASK BIT_MASK(11) #define VPU_HW_BTRS_LNL_VPU_STATUS_DISABLE_CLK_RELINQUISH_MASK BIT_MASK(12) +#define VPU_HW_BTRS_LNL_VPU_STATUS_PLATFORM_MASK GENMASK(31, 29) #define VPU_HW_BTRS_LNL_IP_RESET 0x00000160u #define VPU_HW_BTRS_LNL_IP_RESET_TRIGGER_MASK BIT_MASK(0) From patchwork Tue Jan 7 17:32:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maciej Falkowski X-Patchwork-Id: 13929038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 575DCE77198 for ; Tue, 7 Jan 2025 14:22:56 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C8A1110EAD2; Tue, 7 Jan 2025 14:22:55 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="nFcIV8Z6"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.19]) by gabe.freedesktop.org (Postfix) with ESMTPS id BD8DB10EAD2 for ; Tue, 7 Jan 2025 14:22:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1736259774; x=1767795774; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=J4Qd/MRhohSbcUGpHV4gNjm1mS/1p+HTffB0fCrZ5Tk=; b=nFcIV8Z6VWJOYFpSYEq0JdvowzTBNBGsGI+W3zJFxIbuxVI0lmzo9y4p t8EOQ78GXmterhUgJgwv/tc26SQrHKBoRimReNRBOLt5ejGDq2POBXE/d NpCxtE+2t3YvbqXuPbFno7+scfG9O0jU9XAkLTmG51gv4BhdnviN03ttL ++u+zp+66WMccYLPBIqvLjKimgxv047ZdQvUh5TkLG7OwTGngDcAV9cmy 9mB62bW5PIOorCvuwnOo9vKgTTYw+15qg6zZDJtP4dm7v2tscCJARTfWt GVlH1PNOA+ZuXr0JqRb0tj3wBVgKyivCFBvsrQ/20V0+TjT2vnrL+0tfy A==; X-CSE-ConnectionGUID: ru++RkoBS32wgC3KKdgKiw== X-CSE-MsgGUID: WRt1NPuyTNmrX5B/cFGarg== X-IronPort-AV: E=McAfee;i="6700,10204,11308"; a="36324484" X-IronPort-AV: E=Sophos;i="6.12,295,1728975600"; d="scan'208";a="36324484" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa111.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:54 -0800 X-CSE-ConnectionGUID: RzMphEGcRjeYNCyPNxIO1A== X-CSE-MsgGUID: p7dLvXYPRfCD+hcCh2EWVg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="103635520" Received: from try2-8594.igk.intel.com ([10.91.220.58]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jan 2025 06:22:53 -0800 From: Maciej Falkowski To: dri-devel@lists.freedesktop.org Cc: oded.gabbay@gmail.com, quic_jhugo@quicinc.com, jacek.lawrynowicz@linux.intel.com, Maciej Falkowski Subject: [PATCH 14/14] accel/ivpu: Enable HWS by default on all platforms Date: Tue, 7 Jan 2025 18:32:37 +0100 Message-ID: <20250107173238.381120-15-maciej.falkowski@linux.intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> References: <20250107173238.381120-1-maciej.falkowski@linux.intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Jacek Lawrynowicz Enable HWS on selected platforms if FW API version is above 3.19. Signed-off-by: Karol Wachowski Signed-off-by: Jacek Lawrynowicz Signed-off-by: Maciej Falkowski --- drivers/accel/ivpu/ivpu_fw.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/accel/ivpu/ivpu_fw.c b/drivers/accel/ivpu/ivpu_fw.c index 6037ec0b3096..6cf1fb826d1b 100644 --- a/drivers/accel/ivpu/ivpu_fw.c +++ b/drivers/accel/ivpu/ivpu_fw.c @@ -145,7 +145,18 @@ ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_he if (ivpu_sched_mode != IVPU_SCHED_MODE_AUTO) return ivpu_sched_mode; - return VPU_SCHEDULING_MODE_OS; + if (IVPU_FW_CHECK_API_VER_LT(vdev, fw_hdr, JSM, 3, 24)) + return VPU_SCHEDULING_MODE_OS; + + switch (ivpu_device_id(vdev)) { + case PCI_DEVICE_ID_MTL: + case PCI_DEVICE_ID_ARL: + case PCI_DEVICE_ID_LNL: + case PCI_DEVICE_ID_PTL_P: + return VPU_SCHEDULING_MODE_HW; + default: + return VPU_SCHEDULING_MODE_OS; + } } static int ivpu_fw_parse(struct ivpu_device *vdev)