From patchwork Sat Nov 9 17:29:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13869683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 290E2D5E376 for ; Sat, 9 Nov 2024 17:29:16 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 26C9C10E358; Sat, 9 Nov 2024 17:29:15 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="NIfcRLkO"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4550510E358; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Uz5kzDi9IAxZrVKYL/re+EaFsCe6g7nbZcl0A58Os7E=; b=NIfcRLkONRIDd8lWIaqVNFw0xoi8CNO5mCeejwKg+CJIxNQJEgDiMflp tva1/SW3tKQS5sykCiJV6N1t9ikUaT2z/WI7dQTIU03J4AaBKjXTqxDNt J1hsl5ziS3+MbKQ1B6p4LCH0Fjl09IuPrZ2jFU12EsX0QOCjclUgucXc9 krJQA560Ja4tzFCDwrV43aDXPQxJq5jRmqVTqpH98K8plrpHU3dcECZpc ceTcaxR0O85aQ9vqUh1BaYsCbrpxNAebc3L6SQRvu6TH2sHfnO1kqLnet FP1Pygy5DC6tUC8vmVm1dbBnX5dWlGUJi/P1n6CKbrztTP5y5vqWyTbJS A==; X-CSE-ConnectionGUID: cdrL2O3aT5KAiog3uVNZYQ== X-CSE-MsgGUID: CeudMJmDQBirifiCsdBznA== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600106" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600106" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:13 -0800 X-CSE-ConnectionGUID: EEViczvuQU+4QVObovza8g== X-CSE-MsgGUID: qN0rFfPbS4mJ2iailITIfQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499487" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:12 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 1/6] dma-resv: Add DMA_RESV_USAGE_PREEMPT Date: Sat, 9 Nov 2024 09:29:37 -0800 Message-Id: <20241109172942.482630-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add a new DMA reservation slot, which is the highest priority and used exclusively for preemptive fences. The semantics of this slot require that all other fences in other slots must be signaled before any fences in the preemptive slot enable signaling. These semantics exist to avoid deadlocks during user submissions when using DMA fences. These rules are baked into dma-resv. Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/dma-buf/dma-resv.c | 43 +++++++++++++++++++++++------------ drivers/dma-buf/st-dma-resv.c | 2 +- include/linux/dma-fence.h | 1 + include/linux/dma-resv.h | 24 ++++++++++++++----- 4 files changed, 49 insertions(+), 21 deletions(-) diff --git a/drivers/dma-buf/dma-resv.c b/drivers/dma-buf/dma-resv.c index 5f8d010516f0..7358aa466c75 100644 --- a/drivers/dma-buf/dma-resv.c +++ b/drivers/dma-buf/dma-resv.c @@ -57,9 +57,6 @@ DEFINE_WD_CLASS(reservation_ww_class); EXPORT_SYMBOL(reservation_ww_class); -/* Mask for the lower fence pointer bits */ -#define DMA_RESV_LIST_MASK 0x3 - struct dma_resv_list { struct rcu_head rcu; u32 num_fences, max_fences; @@ -71,13 +68,14 @@ static void dma_resv_list_entry(struct dma_resv_list *list, unsigned int index, struct dma_resv *resv, struct dma_fence **fence, enum dma_resv_usage *usage) { - long tmp; - tmp = (long)rcu_dereference_check(list->table[index], - resv ? dma_resv_held(resv) : true); - *fence = (struct dma_fence *)(tmp & ~DMA_RESV_LIST_MASK); + *fence = (struct dma_fence *)rcu_dereference_check(list->table[index], + resv ? + dma_resv_held(resv) : + true); + if (usage) - *usage = tmp & DMA_RESV_LIST_MASK; + *usage = (*fence)->usage; } /* Set the fence and usage flags at the specific index in the list. */ @@ -86,9 +84,8 @@ static void dma_resv_list_set(struct dma_resv_list *list, struct dma_fence *fence, enum dma_resv_usage usage) { - long tmp = ((long)fence) | usage; - - RCU_INIT_POINTER(list->table[index], (struct dma_fence *)tmp); + fence->usage = usage; + RCU_INIT_POINTER(list->table[index], fence); } /* @@ -527,7 +524,7 @@ int dma_resv_copy_fences(struct dma_resv *dst, struct dma_resv *src) list = NULL; - dma_resv_iter_begin(&cursor, src, DMA_RESV_USAGE_BOOKKEEP); + dma_resv_iter_begin(&cursor, src, DMA_RESV_USAGE_PREEMPT); dma_resv_for_each_fence_unlocked(&cursor, f) { if (dma_resv_iter_is_restarted(&cursor)) { @@ -680,8 +677,11 @@ long dma_resv_wait_timeout(struct dma_resv *obj, enum dma_resv_usage usage, long ret = timeout ? timeout : 1; struct dma_resv_iter cursor; struct dma_fence *fence; + enum dma_resv_usage walk_usage = (usage == DMA_RESV_USAGE_PREEMPT) ? + DMA_RESV_USAGE_BOOKKEEP : usage; - dma_resv_iter_begin(&cursor, obj, usage); +wait_again: + dma_resv_iter_begin(&cursor, obj, walk_usage); dma_resv_for_each_fence_unlocked(&cursor, fence) { ret = dma_fence_wait_timeout(fence, intr, ret); @@ -692,6 +692,20 @@ long dma_resv_wait_timeout(struct dma_resv *obj, enum dma_resv_usage usage, } dma_resv_iter_end(&cursor); + /* + * Now wait on preempt fences, pipeline preemption by enabling signaling + * before waiting. + */ + if (walk_usage != usage) { + dma_resv_iter_begin(&cursor, obj, usage); + dma_resv_for_each_fence_unlocked(&cursor, fence) + dma_fence_enable_sw_signaling(fence); + dma_resv_iter_end(&cursor); + + walk_usage = usage; + goto wait_again; + } + return ret; } EXPORT_SYMBOL_GPL(dma_resv_wait_timeout); @@ -757,7 +771,8 @@ EXPORT_SYMBOL_GPL(dma_resv_test_signaled); */ void dma_resv_describe(struct dma_resv *obj, struct seq_file *seq) { - static const char *usage[] = { "kernel", "write", "read", "bookkeep" }; + static const char *usage[] = { "kernel", "write", "read", "bookkeep", + "preempt" }; struct dma_resv_iter cursor; struct dma_fence *fence; diff --git a/drivers/dma-buf/st-dma-resv.c b/drivers/dma-buf/st-dma-resv.c index 15dbea1462ed..ddef3c99885f 100644 --- a/drivers/dma-buf/st-dma-resv.c +++ b/drivers/dma-buf/st-dma-resv.c @@ -306,7 +306,7 @@ int dma_resv(void) int r; spin_lock_init(&fence_lock); - for (usage = DMA_RESV_USAGE_KERNEL; usage <= DMA_RESV_USAGE_BOOKKEEP; + for (usage = DMA_RESV_USAGE_KERNEL; usage <= DMA_RESV_USAGE_PREEMPT; ++usage) { r = subtests(tests, (void *)(unsigned long)usage); if (r) diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index e7ad819962e3..05f3ec2adece 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -94,6 +94,7 @@ struct dma_fence { unsigned long flags; struct kref refcount; int error; + u8 usage; }; enum dma_fence_flag_bits { diff --git a/include/linux/dma-resv.h b/include/linux/dma-resv.h index c5ab6fd9ebe8..eb40c526d570 100644 --- a/include/linux/dma-resv.h +++ b/include/linux/dma-resv.h @@ -55,9 +55,10 @@ struct dma_resv_list; * This enum describes the different use cases for a dma_resv object and * controls which fences are returned when queried. * - * An important fact is that there is the order KERNEL X-Patchwork-Id: 13869687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA4EAD5E379 for ; Sat, 9 Nov 2024 17:29:23 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 83C5B10E363; Sat, 9 Nov 2024 17:29:15 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Ya4mmyV5"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id 547FE10E359; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pVOl7WNTOsrMVXeRIMy6wmjwXhzwv/z4G4PCD14dnh4=; b=Ya4mmyV53NACYwZWCA7ARS3dMEQNGIFdsJ/QqGh2puqZ6EOHdqX2XF8E F641yhQWn5JhSuQWhgFR9YTk3vUmT2gn963mvsZ4FnxLqjMZYdtcvrxzX ICObVqESxQSgSOt+vmTf14mgGPhbLf0ig0flbiHK/4a1EwIrWb3AjTC0o 3X5EpESpcg9SiqhjQNv+O0mJWaUcTK+r3vhr0gi8eiaiGU7zyWwtsutlQ oQFNZfjOVZaxGAkNMRjuRdb5cXn27Zv1HaDqqat1kckmSeEgpUyZd8ZBi VXENdAsGtXdf9ZQvnKdoPUYo5gjWKZgwSus8h0opDUOemgWMShY6ZMxM8 A==; X-CSE-ConnectionGUID: KDblF98XR2Kf95Oqktraqw== X-CSE-MsgGUID: 5FlDiwCFSw2yBZQbHepYfA== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600114" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600114" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:14 -0800 X-CSE-ConnectionGUID: 95whTxFXThKZu6vzhGk3eg== X-CSE-MsgGUID: /F6FtNB2T5+BIaR1IaN66A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499488" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:13 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 2/6] drm/sched: Teach scheduler about DMA_RESV_USAGE_PREEMPT Date: Sat, 9 Nov 2024 09:29:38 -0800 Message-Id: <20241109172942.482630-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Follow the semantics of DMA_RESV_USAGE_PREEMPT in the DRM scheduler by storing preemptive fences in a dedicated xarray, which is waited on after all other fences are signaled. In addition to following these semantics, pipeline preemptive fences by enabling signaling on all preemptive fences before waiting on any of them. Cc: Philipp Stanner Cc: Danilo Krummrich Cc: Luben Tuikov Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_entity.c | 29 ++++++++++++-- drivers/gpu/drm/scheduler/sched_main.c | 48 ++++++++++++++++-------- include/drm/gpu_scheduler.h | 15 ++++++++ 3 files changed, 73 insertions(+), 19 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 69bcf0e99d57..c6c4978aa65a 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -201,11 +201,13 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, struct drm_sched_job *job = container_of(cb, struct drm_sched_job, finish_cb); unsigned long index; + struct xarray *dependencies = &job->dependencies; dma_fence_put(f); +again: /* Wait for all dependencies to avoid data corruptions */ - xa_for_each(&job->dependencies, index, f) { + xa_for_each(dependencies, index, f) { struct drm_sched_fence *s_fence = to_drm_sched_fence(f); if (s_fence && f == &s_fence->scheduled) { @@ -223,7 +225,7 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, dma_fence_put(&s_fence->scheduled); } - xa_erase(&job->dependencies, index); + xa_erase(dependencies, index); if (f && !dma_fence_add_callback(f, &job->finish_cb, drm_sched_entity_kill_jobs_cb)) return; @@ -231,6 +233,11 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, dma_fence_put(f); } + if (dependencies != &job->preempt_dependencies) { + dependencies = &job->preempt_dependencies; + goto again; + } + INIT_WORK(&job->work, drm_sched_entity_kill_jobs_work); schedule_work(&job->work); } @@ -456,17 +463,33 @@ drm_sched_job_dependency(struct drm_sched_job *job, struct drm_sched_entity *entity) { struct dma_fence *f; + struct xarray *dependencies; + +again: + dependencies = job->resolve_preempt_dependencies ? + &job->preempt_dependencies : &job->dependencies; /* We keep the fence around, so we can iterate over all dependencies * in drm_sched_entity_kill_jobs_cb() to ensure all deps are signaled * before killing the job. */ - f = xa_load(&job->dependencies, job->last_dependency); + f = xa_load(dependencies, job->last_dependency); if (f) { job->last_dependency++; return dma_fence_get(f); } + /* Switch resolving preempt dependencies pipelining signaling */ + if (!job->resolve_preempt_dependencies) { + unsigned long index; + + xa_for_each(&job->preempt_dependencies, index, f) + dma_fence_enable_sw_signaling(f); + + job->resolve_preempt_dependencies = true; + goto again; + } + if (job->sched->ops->prepare_job) return job->sched->ops->prepare_job(job, entity); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 7ce25281c74c..eceb9b8c6f5f 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -829,6 +829,7 @@ int drm_sched_job_init(struct drm_sched_job *job, INIT_LIST_HEAD(&job->list); xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC); + xa_init_flags(&job->preempt_dependencies, XA_FLAGS_ALLOC); return 0; } @@ -864,21 +865,14 @@ void drm_sched_job_arm(struct drm_sched_job *job) } EXPORT_SYMBOL(drm_sched_job_arm); -/** - * drm_sched_job_add_dependency - adds the fence as a job dependency - * @job: scheduler job to add the dependencies to - * @fence: the dma_fence to add to the list of dependencies. - * - * Note that @fence is consumed in both the success and error cases. - * - * Returns: - * 0 on success, or an error on failing to expand the array. - */ -int drm_sched_job_add_dependency(struct drm_sched_job *job, - struct dma_fence *fence) +static int __drm_sched_job_add_dependency(struct drm_sched_job *job, + struct dma_fence *fence, + bool is_preempt) { struct dma_fence *entry; unsigned long index; + struct xarray *dependencies = is_preempt ? &job->preempt_dependencies : + &job->dependencies; u32 id = 0; int ret; @@ -889,25 +883,41 @@ int drm_sched_job_add_dependency(struct drm_sched_job *job, * This lets the size of the array of deps scale with the number of * engines involved, rather than the number of BOs. */ - xa_for_each(&job->dependencies, index, entry) { + xa_for_each(dependencies, index, entry) { if (entry->context != fence->context) continue; if (dma_fence_is_later(fence, entry)) { dma_fence_put(entry); - xa_store(&job->dependencies, index, fence, GFP_KERNEL); + xa_store(dependencies, index, fence, GFP_KERNEL); } else { dma_fence_put(fence); } return 0; } - ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); + ret = xa_alloc(dependencies, &id, fence, xa_limit_32b, GFP_KERNEL); if (ret != 0) dma_fence_put(fence); return ret; } + +/** + * drm_sched_job_add_dependency - adds the fence as a job dependency + * @job: scheduler job to add the dependencies to + * @fence: the dma_fence to add to the list of dependencies. + * + * Note that @fence is consumed in both the success and error cases. + * + * Returns: + * 0 on success, or an error on failing to expand the array. + */ +int drm_sched_job_add_dependency(struct drm_sched_job *job, + struct dma_fence *fence) +{ + return __drm_sched_job_add_dependency(job, fence, false); +} EXPORT_SYMBOL(drm_sched_job_add_dependency); /** @@ -963,7 +973,9 @@ int drm_sched_job_add_resv_dependencies(struct drm_sched_job *job, dma_resv_for_each_fence(&cursor, resv, usage, fence) { /* Make sure to grab an additional ref on the added fence */ dma_fence_get(fence); - ret = drm_sched_job_add_dependency(job, fence); + ret = __drm_sched_job_add_dependency(job, fence, + cursor.fence_usage == + DMA_RESV_USAGE_PREEMPT); if (ret) { dma_fence_put(fence); return ret; @@ -1030,6 +1042,10 @@ void drm_sched_job_cleanup(struct drm_sched_job *job) } xa_destroy(&job->dependencies); + xa_for_each(&job->preempt_dependencies, index, fence) { + dma_fence_put(fence); + } + xa_destroy(&job->preempt_dependencies); } EXPORT_SYMBOL(drm_sched_job_cleanup); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 95e17504e46a..de16cf6b1869 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -353,6 +353,13 @@ struct drm_sched_job { u32 credits; + /** + * @resolve_preempt_dependencies: + * + * Job is currently resolving preempt dependencies. + */ + bool resolve_preempt_dependencies; + /* * work is used only after finish_cb has been used and will not be * accessed anymore. @@ -376,6 +383,14 @@ struct drm_sched_job { */ struct xarray dependencies; + /** + * @preempt_dependencies: + * + * Contains the dependencies as struct dma_fence for this job which are + * preempt fences. + */ + struct xarray preempt_dependencies; + /** @last_dependency: tracks @dependencies as they signal */ unsigned long last_dependency; From patchwork Sat Nov 9 17:29:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13869685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1875CD5E37D for ; Sat, 9 Nov 2024 17:29:21 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5FCE610E35C; Sat, 9 Nov 2024 17:29:15 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="ISmOcBsu"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5A6D510E35B; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=g3Fnr+eeiYufPr352u35VGvUGKDvw6+Cf03KpdN8IJY=; b=ISmOcBsuRwkdlGkJSGl9FGa45vquLKx6Sd3LrTZvgxM+m0s58r8ikfn+ fUylGgmD3ejenkucLAKbOZQfHyj2mfak+iieSc9wZVZ/FNVKlJ36KkCy8 +u4I4x38anQWRxzgvxFboZsacnNBjAICo05hjF6uHTBbxgugIcOEN5GBs KiRTPU9GS+GF8tL3sfPKCd5WpfWMF8S2cnIi8jgsbt5Znuuw94iHqIlYX UJD71xfZM+vtA0A3htA4z/8S8E3PLsEoi6a6RKn5prcOCLmzXnPYa8i1J olZX6yspRoNOHtxev4lCIsLzEtS5gqMD4u8Svlqy4KQQVycEwpvvG+Nkc A==; X-CSE-ConnectionGUID: 0BJvhSRjRxmRgrqK7qsrPQ== X-CSE-MsgGUID: lyxwaL8LRKqlJ+yDv6wj4w== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600123" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600123" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:14 -0800 X-CSE-ConnectionGUID: TlX3u9tRSOKmHJeK7vU7IQ== X-CSE-MsgGUID: lYXUv1+tQ5y5VNPlv5SiNA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499490" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:13 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 3/6] dma-fence: Add dma_fence_preempt base class Date: Sat, 9 Nov 2024 09:29:39 -0800 Message-Id: <20241109172942.482630-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add a dma_fence_preempt base class with driver ops to implement preemption, based on the existing Xe preemptive fence implementation. Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-fence-preempt.c | 102 ++++++++++++++++++++++++++++ include/linux/dma-fence-preempt.h | 54 +++++++++++++++ 3 files changed, 157 insertions(+), 1 deletion(-) create mode 100644 drivers/dma-buf/dma-fence-preempt.c create mode 100644 include/linux/dma-fence-preempt.h diff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile index 70ec901edf2c..c25500bb38b5 100644 --- a/drivers/dma-buf/Makefile +++ b/drivers/dma-buf/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y := dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o + dma-fence-preempt.o dma-fence-unwrap.o dma-resv.o obj-$(CONFIG_DMABUF_HEAPS) += dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) += heaps/ obj-$(CONFIG_SYNC_FILE) += sync_file.o diff --git a/drivers/dma-buf/dma-fence-preempt.c b/drivers/dma-buf/dma-fence-preempt.c new file mode 100644 index 000000000000..e97ddd925db6 --- /dev/null +++ b/drivers/dma-buf/dma-fence-preempt.c @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2024 Intel Corporation + */ + +#include +#include + +static const char * +dma_fence_preempt_get_driver_name(struct dma_fence *fence) +{ + return "dma_fence_preempt"; +} + +static const char * +dma_fence_preempt_get_timeline_name(struct dma_fence *fence) +{ + return "ordered"; +} + +static bool dma_fence_preempt_enable_signaling(struct dma_fence *fence) +{ + struct dma_fence_preempt *pfence = + container_of(fence, typeof(*pfence), base); + int err; + + err = pfence->ops->preempt(pfence); + if (err) + dma_fence_set_error(&pfence->base, err); + else + WARN_ON(!dma_resv_test_signaled(pfence->resv, + DMA_RESV_USAGE_BOOKKEEP)); + + dma_fence_get(fence); + queue_work(pfence->wq, &pfence->work); + + return true; +} + +static const struct dma_fence_ops preempt_fence_ops = { + .get_driver_name = dma_fence_preempt_get_driver_name, + .get_timeline_name = dma_fence_preempt_get_timeline_name, + .enable_signaling = dma_fence_preempt_enable_signaling, +}; + +static void dma_fence_preempt_work_func(struct work_struct *w) +{ + bool cookie = dma_fence_begin_signalling(); + struct dma_fence_preempt *pfence = + container_of(w, typeof(*pfence), work); + int err = pfence->base.error; + + if (!err) { + err = pfence->ops->preempt_wait(pfence); + if (err) + dma_fence_set_error(&pfence->base, err); + } + + dma_fence_signal(&pfence->base); + pfence->ops->preempt_finished(pfence); + + dma_fence_end_signalling(cookie); + dma_fence_put(&pfence->base); +} + +/** + * dma_fence_is_preempt() - Is preempt fence + * + * @fence: Preempt fence + * + * Return: True if preempt fence, False otherwise + */ +bool dma_fence_is_preempt(const struct dma_fence *fence) +{ + return fence->ops == &preempt_fence_ops; +} +EXPORT_SYMBOL(dma_fence_is_preempt); + +/** + * dma_fence_preempt_init() - Initial preempt fence + * + * @fence: Preempt fence + * @ops: Preempt fence operations + * @resv: Dma resv which preempt fence is attached to + * @wq: Work queue for preempt wait + * @context: Fence context + * @seqno: Fence seqence number + */ +void dma_fence_preempt_init(struct dma_fence_preempt *fence, + const struct dma_fence_preempt_ops *ops, + struct dma_resv *resv, struct workqueue_struct *wq, + u64 context, u64 seqno) +{ + fence->ops = ops; + fence->resv = resv; + fence->wq = wq; + INIT_WORK(&fence->work, dma_fence_preempt_work_func); + spin_lock_init(&fence->lock); + dma_fence_init(&fence->base, &preempt_fence_ops, + &fence->lock, context, seqno); +} +EXPORT_SYMBOL(dma_fence_preempt_init); diff --git a/include/linux/dma-fence-preempt.h b/include/linux/dma-fence-preempt.h new file mode 100644 index 000000000000..9fdfe4a6b00f --- /dev/null +++ b/include/linux/dma-fence-preempt.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2024 Intel Corporation + */ + +#ifndef __LINUX_DMA_FENCE_PREEMPT_H +#define __LINUX_DMA_FENCE_PREEMPT_H + +#include +#include + +struct dma_fence_preempt; +struct dma_resv; + +/** + * struct dma_fence_preempt_ops - Preempt fence operations + * + * These functions should be implemented in the driver side. + */ +struct dma_fence_preempt_ops { + /** @preempt: Preempt execution */ + int (*preempt)(struct dma_fence_preempt *fence); + /** @preempt_wait: Wait for preempt of execution to complete */ + int (*preempt_wait)(struct dma_fence_preempt *fence); + /** @preempt_finished: Signal that the preempt has finished */ + void (*preempt_finished)(struct dma_fence_preempt *fence); +}; + +/** + * struct dma_fence_preempt - Embedded preempt fence base class + */ +struct dma_fence_preempt { + /** @base: Fence base class */ + struct dma_fence base; + /** @lock: Spinlock for fence handling */ + spinlock_t lock; + /** @ops: Preempt fence operation */ + const struct dma_fence_preempt_ops *ops; + /** @resv: DMA resv which preempt fence attached to */ + struct dma_resv *resv; + /** @wq: Work queue for preempt wait */ + struct workqueue_struct *wq; + /** @work: Work struct for preempt wait */ + struct work_struct work; +}; + +bool dma_fence_is_preempt(const struct dma_fence *fence); + +void dma_fence_preempt_init(struct dma_fence_preempt *fence, + const struct dma_fence_preempt_ops *ops, + struct dma_resv *resv, struct workqueue_struct *wq, + u64 context, u64 seqno); + +#endif From patchwork Sat Nov 9 17:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13869686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C583D5E37E for ; Sat, 9 Nov 2024 17:29:22 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7DA7010E360; Sat, 9 Nov 2024 17:29:15 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="BaeAv+tN"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6334E10E357; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jL521Rg11rGFLjmPgqAuPSkCfXrq6c598uHpSXKSI6U=; b=BaeAv+tN6uvMvgD0CAUsC+3AriKDcCVqm0fgiCi+4YXzMWACjN2zq5Mr Z/LMR7kqIz++v1JqzHbxWlt7WwK/GZ5h/6ERdPVmYZ6Oa1tal6BHi1W/o zH6t0q9CqaUpDSEoCvrPpC1F3V2o8E9f0L4Ztn4YNp86RjyKpZc3B1M2l JkDPWbZyXh/RwByqKsOSEF8I6Xsh8dGYQnkOlw/28UpU2g3OZrsFRUEGK MZDsayTDlL9bCXIsAE32bcFomv311CO9PIXo64g+O3zoj3TxF7mpTOm+f GJdf6moX4K1yXMSQOvX24RkEDj2GXE9iysQ7X3v9hnqp5QdP8VLEHZJgA g==; X-CSE-ConnectionGUID: 26s3niMrQ062fSqSaipUWw== X-CSE-MsgGUID: Xmoz32DJQR2TRgXlzNjDAg== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600130" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600130" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:15 -0800 X-CSE-ConnectionGUID: yC+fD0VWRWe4IDrnMKM6Rg== X-CSE-MsgGUID: H35kI1fURzStITDrHCvkOQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499491" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:13 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 4/6] drm/sched: Teach scheduler about dma_fence_prempt type Date: Sat, 9 Nov 2024 09:29:40 -0800 Message-Id: <20241109172942.482630-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Update drm_sched_job_add_dependency to detect dma_fence_preempt and correctly store it in the dedicated preemptive fence xarray ensuring preempt fence semantics are followed. Cc: Philipp Stanner Cc: Danilo Krummrich Cc: Luben Tuikov Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index eceb9b8c6f5f..607722364876 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -76,6 +76,7 @@ #include #include #include +#include #include #include @@ -916,7 +917,8 @@ static int __drm_sched_job_add_dependency(struct drm_sched_job *job, int drm_sched_job_add_dependency(struct drm_sched_job *job, struct dma_fence *fence) { - return __drm_sched_job_add_dependency(job, fence, false); + return __drm_sched_job_add_dependency(job, fence, + dma_fence_is_preempt(fence)); } EXPORT_SYMBOL(drm_sched_job_add_dependency); From patchwork Sat Nov 9 17:29:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13869689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 864D9D5E37A for ; Sat, 9 Nov 2024 17:29:25 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 045C510E36E; Sat, 9 Nov 2024 17:29:17 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="Ka5T3NzY"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id AFACB10E357; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AoY1SdHGeG5w1ptfOQPooNP7s2JrJ8K10jQpyOgPu0k=; b=Ka5T3NzYyY8U4ZXmCPwDkLfbl7GUXVwnnER39sSICx+W0lxAzW95PEzy LfjFm5ICTAXCJCFCq/77BrwcaC1XmUdCe3smveMevoDiNE4tc7BL7QQSj hk5pwHjXxy1rIu8LNR5OcowXTmjOXkrKaKNkJVtvQsBX12TZuTBiA7Q78 rR2asHiWVxZ+2ZTmniS4P+OrezQ2MHgCDt+xMKmMznNGeN2EZfSNcumhK hesLnR6VxGNyakXVHT15TqevicoJ60WY57bMRvUZlA8FhF9u7aLwYO4u8 py/8OdycTSSV8k8WTItmO5Yu37n2hyLqfPxJZi5zvZ72YPwXsvxEdiqsQ g==; X-CSE-ConnectionGUID: y/F6fdTsTUmgmEzxbIAspA== X-CSE-MsgGUID: Lgb3hap9QLi7MM8RlRPuSg== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600139" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600139" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:15 -0800 X-CSE-ConnectionGUID: iSHUwmOKQPqD0pTIxDlVEA== X-CSE-MsgGUID: SHyQiTRDQA6C7DbCYqbGbQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499492" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:14 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 5/6] drm/xe: Use DMA_RESV_USAGE_PREEMPT for preempt fences Date: Sat, 9 Nov 2024 09:29:41 -0800 Message-Id: <20241109172942.482630-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Use the new DMA_RESV_USAGE_PREEMPT dma-resv slots in Xe for preemptive fences, and call them in dma-resv/scheduler rather than open-coding the enabling of signaling before waiting. Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_bo.c | 22 +++++----------------- drivers/gpu/drm/xe/xe_migrate.c | 4 ++-- drivers/gpu/drm/xe/xe_pt.c | 12 ++---------- drivers/gpu/drm/xe/xe_vm.c | 22 ++++------------------ 4 files changed, 13 insertions(+), 47 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c index 549866da5cd1..7ab7d27b66c6 100644 --- a/drivers/gpu/drm/xe/xe_bo.c +++ b/drivers/gpu/drm/xe/xe_bo.c @@ -484,23 +484,11 @@ static int xe_ttm_io_mem_reserve(struct ttm_device *bdev, static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, const struct ttm_operation_ctx *ctx) { - struct dma_resv_iter cursor; - struct dma_fence *fence; struct drm_gem_object *obj = &bo->ttm.base; struct drm_gpuvm_bo *vm_bo; bool idle = false; int ret = 0; - dma_resv_assert_held(bo->ttm.base.resv); - - if (!list_empty(&bo->ttm.base.gpuva.list)) { - dma_resv_iter_begin(&cursor, bo->ttm.base.resv, - DMA_RESV_USAGE_BOOKKEEP); - dma_resv_for_each_fence_unlocked(&cursor, fence) - dma_fence_enable_sw_signaling(fence); - dma_resv_iter_end(&cursor); - } - drm_gem_for_each_gpuvm_bo(vm_bo, obj) { struct xe_vm *vm = gpuvm_to_vm(vm_bo->vm); struct drm_gpuva *gpuva; @@ -515,11 +503,11 @@ static int xe_bo_trigger_rebind(struct xe_device *xe, struct xe_bo *bo, if (ctx->no_wait_gpu && !dma_resv_test_signaled(bo->ttm.base.resv, - DMA_RESV_USAGE_BOOKKEEP)) + DMA_RESV_USAGE_PREEMPT)) return -EBUSY; timeout = dma_resv_wait_timeout(bo->ttm.base.resv, - DMA_RESV_USAGE_BOOKKEEP, + DMA_RESV_USAGE_PREEMPT, ctx->interruptible, MAX_SCHEDULE_TIMEOUT); if (!timeout) @@ -723,7 +711,7 @@ static int xe_bo_move(struct ttm_buffer_object *ttm_bo, bool evict, if (old_mem_type == XE_PL_TT && new_mem->mem_type == XE_PL_SYSTEM) { long timeout = dma_resv_wait_timeout(ttm_bo->base.resv, - DMA_RESV_USAGE_BOOKKEEP, + DMA_RESV_USAGE_PREEMPT, true, MAX_SCHEDULE_TIMEOUT); if (timeout < 0) { @@ -1056,7 +1044,7 @@ static void xe_ttm_bo_release_notify(struct ttm_buffer_object *ttm_bo) * unbind. */ dma_resv_for_each_fence(&cursor, ttm_bo->base.resv, - DMA_RESV_USAGE_BOOKKEEP, fence) { + DMA_RESV_USAGE_PREEMPT, fence) { if (xe_fence_is_xe_preempt(fence) && !dma_fence_is_signaled(fence)) { if (!replacement) @@ -1065,7 +1053,7 @@ static void xe_ttm_bo_release_notify(struct ttm_buffer_object *ttm_bo) dma_resv_replace_fences(ttm_bo->base.resv, fence->context, replacement, - DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT); } } dma_fence_put(replacement); diff --git a/drivers/gpu/drm/xe/xe_migrate.c b/drivers/gpu/drm/xe/xe_migrate.c index cfd31ae49cc1..75067c584581 100644 --- a/drivers/gpu/drm/xe/xe_migrate.c +++ b/drivers/gpu/drm/xe/xe_migrate.c @@ -895,10 +895,10 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, xe_sched_job_add_migrate_flush(job, flush_flags); if (!fence) { err = xe_sched_job_add_deps(job, src_bo->ttm.base.resv, - DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT); if (!err && src_bo != dst_bo) err = xe_sched_job_add_deps(job, dst_bo->ttm.base.resv, - DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT); if (err) goto err_job; } diff --git a/drivers/gpu/drm/xe/xe_pt.c b/drivers/gpu/drm/xe/xe_pt.c index f27f579f4d85..00358e748d0d 100644 --- a/drivers/gpu/drm/xe/xe_pt.c +++ b/drivers/gpu/drm/xe/xe_pt.c @@ -1115,7 +1115,7 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job, if (pt_update_ops->wait_vm_bookkeep || pt_update_ops->wait_vm_kernel) { err = job_test_add_deps(job, xe_vm_resv(vm), pt_update_ops->wait_vm_bookkeep ? - DMA_RESV_USAGE_BOOKKEEP : + DMA_RESV_USAGE_PREEMPT : DMA_RESV_USAGE_KERNEL); if (err) return err; @@ -1231,18 +1231,10 @@ static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma, spin_unlock(&vm->userptr.invalidated_lock); if (xe_vm_in_preempt_fence_mode(vm)) { - struct dma_resv_iter cursor; - struct dma_fence *fence; long err; - dma_resv_iter_begin(&cursor, xe_vm_resv(vm), - DMA_RESV_USAGE_BOOKKEEP); - dma_resv_for_each_fence_unlocked(&cursor, fence) - dma_fence_enable_sw_signaling(fence); - dma_resv_iter_end(&cursor); - err = dma_resv_wait_timeout(xe_vm_resv(vm), - DMA_RESV_USAGE_BOOKKEEP, + DMA_RESV_USAGE_PREEMPT, false, MAX_SCHEDULE_TIMEOUT); XE_WARN_ON(err <= 0); } diff --git a/drivers/gpu/drm/xe/xe_vm.c b/drivers/gpu/drm/xe/xe_vm.c index 624133fae5f5..568395530c49 100644 --- a/drivers/gpu/drm/xe/xe_vm.c +++ b/drivers/gpu/drm/xe/xe_vm.c @@ -195,7 +195,7 @@ static int add_preempt_fences(struct xe_vm *vm, struct xe_bo *bo) if (q->lr.pfence) { dma_resv_add_fence(bo->ttm.base.resv, q->lr.pfence, - DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT); } return 0; @@ -213,7 +213,7 @@ static void resume_and_reinstall_preempt_fences(struct xe_vm *vm, q->ops->resume(q); drm_gpuvm_resv_add_fence(&vm->gpuvm, exec, q->lr.pfence, - DMA_RESV_USAGE_BOOKKEEP, DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT, DMA_RESV_USAGE_PREEMPT); } } @@ -250,7 +250,7 @@ int xe_vm_add_compute_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q) down_read(&vm->userptr.notifier_lock); drm_gpuvm_resv_add_fence(&vm->gpuvm, exec, pfence, - DMA_RESV_USAGE_BOOKKEEP, DMA_RESV_USAGE_BOOKKEEP); + DMA_RESV_USAGE_PREEMPT, DMA_RESV_USAGE_PREEMPT); /* * Check to see if a preemption on VM is in flight or userptr @@ -588,8 +588,6 @@ static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni, struct xe_userptr_vma *uvma = container_of(userptr, typeof(*uvma), userptr); struct xe_vma *vma = &uvma->vma; struct xe_vm *vm = xe_vma_vm(vma); - struct dma_resv_iter cursor; - struct dma_fence *fence; long err; xe_assert(vm->xe, xe_vma_is_userptr(vma)); @@ -625,20 +623,8 @@ static bool vma_userptr_invalidate(struct mmu_interval_notifier *mni, up_write(&vm->userptr.notifier_lock); - /* - * Preempt fences turn into schedule disables, pipeline these. - * Note that even in fault mode, we need to wait for binds and - * unbinds to complete, and those are attached as BOOKMARK fences - * to the vm. - */ - dma_resv_iter_begin(&cursor, xe_vm_resv(vm), - DMA_RESV_USAGE_BOOKKEEP); - dma_resv_for_each_fence_unlocked(&cursor, fence) - dma_fence_enable_sw_signaling(fence); - dma_resv_iter_end(&cursor); - err = dma_resv_wait_timeout(xe_vm_resv(vm), - DMA_RESV_USAGE_BOOKKEEP, + DMA_RESV_USAGE_PREEMPT, false, MAX_SCHEDULE_TIMEOUT); XE_WARN_ON(err <= 0); From patchwork Sat Nov 9 17:29:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13869688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD941D5E37B for ; Sat, 9 Nov 2024 17:29:24 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3E6E210E36C; Sat, 9 Nov 2024 17:29:16 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.b="cspT+nq/"; dkim-atps=neutral Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) by gabe.freedesktop.org (Postfix) with ESMTPS id EEEFE10E357; Sat, 9 Nov 2024 17:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1731173355; x=1762709355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=riqz8eY/siyAhQVH1HHpAKOLT3Phpq/w4bOO076Xg54=; b=cspT+nq/VdD1rpbUo0DfwFTim/or0fE+plvy1gm97munVboQ9IfpShj4 6fYJLrQgkB3VMHZws86GXt0VeO72se2PcfD+Kn8Y22BfsL2OwsFJdtn7R LUvrF+rUaWQSkQq7vzK1YjhcUTS2PbrQHvFOzQzvY/8IcPZ5oXz8p5reG b88B5oRbj4aLilnn2jV2h0xdon/In9+GOlAwighPvTZ5GIwK11bcPsp9M 1l2WiDmtqo4GpcXc6+ZDFxeh9knaE6mZRVdCCEmzwZ2YWJYNS8LQQ5F3r /5I2fRSpwCpzueF+qF+wyScGUiWX5V9+MvTxmgOaM0LCHSmWNfsFXOe72 Q==; X-CSE-ConnectionGUID: piS6RrjeSs6U7q1nSgEoHw== X-CSE-MsgGUID: 1yvXiiW7Sry8JcfcFMzaKQ== X-IronPort-AV: E=McAfee;i="6700,10204,11222"; a="41600147" X-IronPort-AV: E=Sophos;i="6.11,199,1725346800"; d="scan'208";a="41600147" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:15 -0800 X-CSE-ConnectionGUID: uvn8IVJhSK6vnjGPd8IE6Q== X-CSE-MsgGUID: JvxwK6OzTAGQ+rmBUxrYdw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,141,1728975600"; d="scan'208";a="86499494" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Nov 2024 09:29:14 -0800 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org Cc: simona.vetter@ffwll.ch, thomas.hellstrom@linux.intel.com, pstanner@redhat.com, boris.brezillon@collabora.com, airlied@gmail.com, ltuikov89@gmail.com, dakr@kernel.org, christian.koenig@amd.com, mihail.atanassov@arm.com, steven.price@arm.com, shashank.sharma@amd.com Subject: [RFC PATCH 6/6] drm/xe: Use dma_fence_preempt base class Date: Sat, 9 Nov 2024 09:29:42 -0800 Message-Id: <20241109172942.482630-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241109172942.482630-1-matthew.brost@intel.com> References: <20241109172942.482630-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Use the dma_fence_preempt base class in Xe instead of open-coding the preemption implementation. Cc: Dave Airlie Cc: Simona Vetter Cc: Christian Koenig Signed-off-by: Matthew Brost --- drivers/gpu/drm/xe/xe_guc_submit.c | 3 + drivers/gpu/drm/xe/xe_hw_engine_group.c | 4 +- drivers/gpu/drm/xe/xe_preempt_fence.c | 81 ++++++--------------- drivers/gpu/drm/xe/xe_preempt_fence.h | 2 +- drivers/gpu/drm/xe/xe_preempt_fence_types.h | 11 +-- 5 files changed, 32 insertions(+), 69 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index 9e0f86f3778b..0411ec9ed705 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -1603,6 +1603,9 @@ static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q) struct xe_guc *guc = exec_queue_to_guc(q); int ret; + if (exec_queue_reset(q) || exec_queue_killed_or_banned_or_wedged(q)) + return -ECANCELED; + /* * Likely don't need to check exec_queue_killed() as we clear * suspend_pending upon kill but to be paranoid but races in which diff --git a/drivers/gpu/drm/xe/xe_hw_engine_group.c b/drivers/gpu/drm/xe/xe_hw_engine_group.c index 82750520a90a..8ed5410c3964 100644 --- a/drivers/gpu/drm/xe/xe_hw_engine_group.c +++ b/drivers/gpu/drm/xe/xe_hw_engine_group.c @@ -163,7 +163,7 @@ int xe_hw_engine_group_add_exec_queue(struct xe_hw_engine_group *group, struct x if (xe_vm_in_fault_mode(q->vm) && group->cur_mode == EXEC_MODE_DMA_FENCE) { q->ops->suspend(q); err = q->ops->suspend_wait(q); - if (err) + if (err == -ETIME) goto err_suspend; xe_hw_engine_group_resume_faulting_lr_jobs(group); @@ -236,7 +236,7 @@ static int xe_hw_engine_group_suspend_faulting_lr_jobs(struct xe_hw_engine_group continue; err = q->ops->suspend_wait(q); - if (err) + if (err == -ETIME) goto err_suspend; } diff --git a/drivers/gpu/drm/xe/xe_preempt_fence.c b/drivers/gpu/drm/xe/xe_preempt_fence.c index 83fbeea5aa20..34418454d65c 100644 --- a/drivers/gpu/drm/xe/xe_preempt_fence.c +++ b/drivers/gpu/drm/xe/xe_preempt_fence.c @@ -4,73 +4,40 @@ */ #include "xe_preempt_fence.h" - -#include - #include "xe_exec_queue.h" #include "xe_vm.h" -static void preempt_fence_work_func(struct work_struct *w) +static struct xe_exec_queue *to_exec_queue(struct dma_fence_preempt *fence) { - bool cookie = dma_fence_begin_signalling(); - struct xe_preempt_fence *pfence = - container_of(w, typeof(*pfence), preempt_work); - struct xe_exec_queue *q = pfence->q; - - if (pfence->error) { - dma_fence_set_error(&pfence->base, pfence->error); - } else if (!q->ops->reset_status(q)) { - int err = q->ops->suspend_wait(q); - - if (err) - dma_fence_set_error(&pfence->base, err); - } else { - dma_fence_set_error(&pfence->base, -ENOENT); - } - - dma_fence_signal(&pfence->base); - /* - * Opt for keep everything in the fence critical section. This looks really strange since we - * have just signalled the fence, however the preempt fences are all signalled via single - * global ordered-wq, therefore anything that happens in this callback can easily block - * progress on the entire wq, which itself may prevent other published preempt fences from - * ever signalling. Therefore try to keep everything here in the callback in the fence - * critical section. For example if something below grabs a scary lock like vm->lock, - * lockdep should complain since we also hold that lock whilst waiting on preempt fences to - * complete. - */ - xe_vm_queue_rebind_worker(q->vm); - xe_exec_queue_put(q); - dma_fence_end_signalling(cookie); + return container_of(fence, struct xe_preempt_fence, base)->q; } -static const char * -preempt_fence_get_driver_name(struct dma_fence *fence) +static int xe_preempt_fence_preempt(struct dma_fence_preempt *fence) { - return "xe"; + struct xe_exec_queue *q = to_exec_queue(fence); + + return q->ops->suspend(q); } -static const char * -preempt_fence_get_timeline_name(struct dma_fence *fence) +static int xe_preempt_fence_preempt_wait(struct dma_fence_preempt *fence) { - return "preempt"; + struct xe_exec_queue *q = to_exec_queue(fence); + + return q->ops->suspend_wait(q); } -static bool preempt_fence_enable_signaling(struct dma_fence *fence) +static void xe_preempt_fence_preempt_finished(struct dma_fence_preempt *fence) { - struct xe_preempt_fence *pfence = - container_of(fence, typeof(*pfence), base); - struct xe_exec_queue *q = pfence->q; + struct xe_exec_queue *q = to_exec_queue(fence); - pfence->error = q->ops->suspend(q); - queue_work(q->vm->xe->preempt_fence_wq, &pfence->preempt_work); - return true; + xe_vm_queue_rebind_worker(q->vm); + xe_exec_queue_put(q); } -static const struct dma_fence_ops preempt_fence_ops = { - .get_driver_name = preempt_fence_get_driver_name, - .get_timeline_name = preempt_fence_get_timeline_name, - .enable_signaling = preempt_fence_enable_signaling, +static const struct dma_fence_preempt_ops xe_preempt_fence_ops = { + .preempt = xe_preempt_fence_preempt, + .preempt_wait = xe_preempt_fence_preempt_wait, + .preempt_finished = xe_preempt_fence_preempt_finished, }; /** @@ -95,7 +62,6 @@ struct xe_preempt_fence *xe_preempt_fence_alloc(void) return ERR_PTR(-ENOMEM); INIT_LIST_HEAD(&pfence->link); - INIT_WORK(&pfence->preempt_work, preempt_fence_work_func); return pfence; } @@ -134,11 +100,12 @@ xe_preempt_fence_arm(struct xe_preempt_fence *pfence, struct xe_exec_queue *q, { list_del_init(&pfence->link); pfence->q = xe_exec_queue_get(q); - spin_lock_init(&pfence->lock); - dma_fence_init(&pfence->base, &preempt_fence_ops, - &pfence->lock, context, seqno); - return &pfence->base; + dma_fence_preempt_init(&pfence->base, &xe_preempt_fence_ops, + xe_vm_resv(q->vm), q->vm->xe->preempt_fence_wq, + context, seqno); + + return &pfence->base.base; } /** @@ -169,5 +136,5 @@ xe_preempt_fence_create(struct xe_exec_queue *q, bool xe_fence_is_xe_preempt(const struct dma_fence *fence) { - return fence->ops == &preempt_fence_ops; + return dma_fence_is_preempt(fence); } diff --git a/drivers/gpu/drm/xe/xe_preempt_fence.h b/drivers/gpu/drm/xe/xe_preempt_fence.h index 9406c6fea525..7b56d12c0786 100644 --- a/drivers/gpu/drm/xe/xe_preempt_fence.h +++ b/drivers/gpu/drm/xe/xe_preempt_fence.h @@ -25,7 +25,7 @@ xe_preempt_fence_arm(struct xe_preempt_fence *pfence, struct xe_exec_queue *q, static inline struct xe_preempt_fence * to_preempt_fence(struct dma_fence *fence) { - return container_of(fence, struct xe_preempt_fence, base); + return container_of(fence, struct xe_preempt_fence, base.base); } /** diff --git a/drivers/gpu/drm/xe/xe_preempt_fence_types.h b/drivers/gpu/drm/xe/xe_preempt_fence_types.h index 312c3372a49f..f12b89f7dc35 100644 --- a/drivers/gpu/drm/xe/xe_preempt_fence_types.h +++ b/drivers/gpu/drm/xe/xe_preempt_fence_types.h @@ -6,8 +6,7 @@ #ifndef _XE_PREEMPT_FENCE_TYPES_H_ #define _XE_PREEMPT_FENCE_TYPES_H_ -#include -#include +#include struct xe_exec_queue; @@ -18,17 +17,11 @@ struct xe_exec_queue; */ struct xe_preempt_fence { /** @base: dma fence base */ - struct dma_fence base; + struct dma_fence_preempt base; /** @link: link into list of pending preempt fences */ struct list_head link; /** @q: exec queue for this preempt fence */ struct xe_exec_queue *q; - /** @preempt_work: work struct which issues preemption */ - struct work_struct preempt_work; - /** @lock: dma-fence fence lock */ - spinlock_t lock; - /** @error: preempt fence is in error state */ - int error; }; #endif