From patchwork Mon Feb 1 08:57:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 12058437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80DF3C43381 for ; Mon, 1 Feb 2021 08:57:59 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2596264E33 for ; Mon, 1 Feb 2021 08:57:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2596264E33 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B576C6E514; Mon, 1 Feb 2021 08:57:38 +0000 (UTC) Received: from fireflyinternet.com (unknown [77.68.26.236]) by gabe.freedesktop.org (Postfix) with ESMTPS id 826F76E4A1 for ; Mon, 1 Feb 2021 08:57:32 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 23757787-1500050 for multiple; Mon, 01 Feb 2021 08:57:25 +0000 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Mon, 1 Feb 2021 08:57:12 +0000 Message-Id: <20210201085715.27435-54-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210201085715.27435-1-chris@chris-wilson.co.uk> References: <20210201085715.27435-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 54/57] drm/i915/gt: Infrastructure for ring scheduling X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Build a bare bones scheduler to sit on top the global legacy ringbuffer submission. This virtual execlists scheme should be applicable to all older platforms. A key problem we have with the legacy ring buffer submission is that it only allows for FIFO queuing. All clients share the global request queue and must contend for its lock when submitting. As any client may need to wait for external events, all clients must then wait. However, if we stage each client into their own virtual ringbuffer with their own timelines, we can copy the client requests into the global ringbuffer only when they are ready, reordering the submission around stalls. Furthermore, the ability to reorder gives us rudimentarily priority sorting -- although without preemption support, once something is on the GPU it stays on the GPU, and so it is still possible for a hog to delay a high priority request (such as updating the display). However, it does means that in keeping a short submission queue, the high priority request will be next. This design resembles the old guc submission scheduler, for reordering requests onto a global workqueue. The implementation uses the MI_USER_INTERRUPT at the end of every request to track completion, so is more interrupt happy than execlists [which has an interrupt for each context event, albeit two]. Our interrupts on these system are relatively heavy, and in the past we have been able to completely starve Sandybrige by the interrupt traffic. Our interrupt handlers are being much better (in part offloading the work to bottom halves leaving the interrupt itself only dealing with acking the registers) but we can still see the impact of starvation in the uneven submission latency on a saturated system. Overall though, the short sumission queues and extra interrupts do not appear to be affecting throughput (+-10%, some tasks even improve to the reduced request overheads) and improve latency. [Which is a massive improvement since the introduction of Sandybridge!] Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_engine_types.h | 1 + .../gpu/drm/i915/gt/intel_ring_scheduler.c | 783 ++++++++++++++++++ .../gpu/drm/i915/gt/intel_ring_submission.c | 17 +- .../gpu/drm/i915/gt/intel_ring_submission.h | 17 + 6 files changed, 812 insertions(+), 8 deletions(-) create mode 100644 drivers/gpu/drm/i915/gt/intel_ring_scheduler.c create mode 100644 drivers/gpu/drm/i915/gt/intel_ring_submission.h diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index ce01634d4ea7..1f9c98eae605 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -115,6 +115,7 @@ gt-y += \ gt/intel_renderstate.o \ gt/intel_reset.o \ gt/intel_ring.o \ + gt/intel_ring_scheduler.o \ gt/intel_ring_submission.o \ gt/intel_rps.o \ gt/intel_sseu.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 33a29623571d..bc07c96ab48c 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -193,6 +193,7 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine); int intel_engine_resume(struct intel_engine_cs *engine); int intel_ring_submission_setup(struct intel_engine_cs *engine); +int intel_ring_scheduler_setup(struct intel_engine_cs *engine); int intel_engine_stop_cs(struct intel_engine_cs *engine); void intel_engine_cancel_stop_cs(struct intel_engine_cs *engine); diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h index e94c99dee5cb..9f14cc631287 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h @@ -318,6 +318,7 @@ struct intel_engine_cs { struct { struct intel_ring *ring; struct intel_timeline *timeline; + struct intel_context *context; } legacy; /* diff --git a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c new file mode 100644 index 000000000000..b6fcb18ef0a6 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c @@ -0,0 +1,783 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include + +#include + +#include "i915_drv.h" +#include "intel_breadcrumbs.h" +#include "intel_context.h" +#include "intel_engine_pm.h" +#include "intel_engine_stats.h" +#include "intel_gt.h" +#include "intel_gt_pm.h" +#include "intel_gt_requests.h" +#include "intel_reset.h" +#include "intel_ring.h" +#include "intel_ring_submission.h" +#include "shmem_utils.h" + +/* + * Rough estimate of the typical request size, performing a flush, + * set-context and then emitting the batch. + */ +#define LEGACY_REQUEST_SIZE 200 + +static void +set_current_context(struct intel_context **ptr, struct intel_context *ce) +{ + if (ce) + intel_context_get(ce); + + ce = xchg(ptr, ce); + + if (ce) + intel_context_put(ce); +} + +static inline void runtime_start(struct intel_context *ce) +{ + struct intel_context_stats *stats = &ce->stats; + + if (intel_context_is_barrier(ce)) + return; + + if (stats->active) + return; + + WRITE_ONCE(stats->active, intel_context_clock()); +} + +static inline void runtime_stop(struct intel_context *ce) +{ + struct intel_context_stats *stats = &ce->stats; + ktime_t dt; + + if (!stats->active) + return; + + dt = ktime_sub(intel_context_clock(), stats->active); + ewma_runtime_add(&stats->runtime.avg, dt); + stats->runtime.total += dt; + + WRITE_ONCE(stats->active, 0); +} + +static struct intel_engine_cs *__schedule_in(struct i915_request *rq) +{ + struct intel_context *ce = rq->context; + struct intel_engine_cs *engine = rq->engine; + + intel_context_get(ce); + + __intel_gt_pm_get(engine->gt); + if (engine->fw_domain && !engine->fw_active++) + intel_uncore_forcewake_get(engine->uncore, engine->fw_domain); + + intel_engine_context_in(engine); + + CE_TRACE(ce, "schedule-in\n"); + + return engine; +} + +static void schedule_in(struct i915_request *rq) +{ + struct intel_context * const ce = rq->context; + struct intel_engine_cs *old; + + GEM_BUG_ON(!intel_engine_pm_is_awake(rq->engine)); + + old = ce->inflight; + if (!old) + old = __schedule_in(rq); + WRITE_ONCE(ce->inflight, ptr_inc(old)); + + GEM_BUG_ON(intel_context_inflight(ce) != rq->engine); + GEM_BUG_ON(!intel_context_inflight_count(ce)); +} + +static void __schedule_out(struct i915_request *rq) +{ + struct intel_context *ce = rq->context; + struct intel_engine_cs *engine = rq->engine; + + CE_TRACE(ce, "schedule-out\n"); + + if (intel_timeline_is_last(ce->timeline, rq)) + intel_engine_add_retire(engine, ce->timeline); + else + i915_request_update_deadline(list_next_entry(rq, link)); + + intel_engine_context_out(engine); + + if (engine->fw_domain && !--engine->fw_active) + intel_uncore_forcewake_put(engine->uncore, engine->fw_domain); + intel_gt_pm_put_async(engine->gt); +} + +static void schedule_out(struct i915_request *rq) +{ + struct intel_context *ce = rq->context; + + GEM_BUG_ON(!ce->inflight); + ce->inflight = ptr_dec(ce->inflight); + if (!intel_context_inflight_count(ce)) { + GEM_BUG_ON(ce->inflight != rq->engine); + __schedule_out(rq); + WRITE_ONCE(ce->inflight, NULL); + intel_context_put(ce); + } + + i915_request_put(rq); +} + +static u32 *ring_map(struct intel_ring *ring, u32 len) +{ + u32 *va; + + if (unlikely(ring->tail + len > ring->effective_size)) { + memset(ring->vaddr + ring->tail, 0, ring->size - ring->tail); + ring->tail = 0; + } + + va = ring->vaddr + ring->tail; + ring->tail = intel_ring_wrap(ring, ring->tail + len); + + return va; +} + +static inline u32 *ring_map_dw(struct intel_ring *ring, u32 len) +{ + return ring_map(ring, len * sizeof(u32)); +} + +static inline void ring_advance(struct intel_ring *ring, void *map) +{ + GEM_BUG_ON(intel_ring_wrap(ring, map - ring->vaddr) != ring->tail); +} + +static void ring_copy(struct intel_ring *dst, + const struct intel_ring *src, + u32 start, u32 end) +{ + unsigned int len; + void *out; + + len = end - start; + if (end < start) + len += src->size; + out = ring_map(dst, len); + + if (end < start) { + len = src->size - start; + memcpy(out, src->vaddr + start, len); + out += len; + start = 0; + } + + memcpy(out, src->vaddr + start, end - start); +} + +static void switch_context(struct intel_ring *ring, struct i915_request *rq) +{ +} + +static struct i915_request *ring_submit(struct i915_request *rq) +{ + struct intel_ring *ring = rq->engine->legacy.ring; + + __i915_request_submit(rq); + + if (rq->engine->legacy.context != rq->context) { + switch_context(ring, rq); + set_current_context(&rq->engine->legacy.context, rq->context); + } + + ring_copy(ring, rq->ring, rq->head, rq->tail); + return rq; +} + +static struct i915_request ** +copy_active(struct i915_request **port, struct i915_request * const *active) +{ + while (*active) + *port++ = *active++; + + return port; +} + +static inline void +copy_ports(struct i915_request **dst, struct i915_request **src, int count) +{ + /* A memcpy_p() would be very useful here! */ + while (count--) + WRITE_ONCE(*dst++, *src++); /* avoid write tearing */ +} + +static inline void write_tail(const struct intel_engine_cs *engine) +{ + wmb(); /* paranoid flush of WCB before RING_TAIL write */ + ENGINE_WRITE(engine, RING_TAIL, engine->legacy.ring->tail); +} + +static void dequeue(struct i915_sched *se, struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request ** const last_port = el->pending + el->port_mask; + struct i915_request **port, **first, *last; + struct i915_priolist *p; + + first = copy_active(el->pending, el->active); + if (first > last_port) + return; + + local_irq_disable(); + + last = NULL; + port = first; + spin_lock(&se->lock); + for_each_priolist(p, &se->queue) { + struct i915_request *rq, *rn; + + priolist_for_each_request_safe(rq, rn, p) { + GEM_BUG_ON(rq == last); + if (last && rq->context != last->context) { + if (port == last_port) + goto done; + + *port++ = i915_request_get(last); + } + + last = ring_submit(rq); + } + + i915_priolist_advance(&se->queue, p); + } +done: + spin_unlock(&se->lock); + + if (last) { + *port++ = i915_request_get(last); + *port = NULL; + + if (!*el->active) + runtime_start((*el->pending)->context); + WRITE_ONCE(el->active, el->pending); + + copy_ports(el->inflight, el->pending, port - el->pending + 1); + while (port-- != first) + schedule_in(*port); + + write_tail(engine); + + WRITE_ONCE(el->active, el->inflight); + GEM_BUG_ON(!*el->active); + } + + WRITE_ONCE(el->pending[0], NULL); + + local_irq_enable(); /* flush irq_work *after* RING_TAIL write */ +} + +static void post_process_csb(struct i915_request **port, + struct i915_request **last) +{ + while (port != last) + schedule_out(*port++); +} + +static struct i915_request ** +process_csb(struct intel_engine_execlists *el, struct i915_request **inactive) +{ + struct i915_request *rq; + + while ((rq = *el->active)) { + if (!__i915_request_is_complete(rq)) { + runtime_start(rq->context); + break; + } + + *inactive++ = rq; + el->active++; + + runtime_stop(rq->context); + } + + return inactive; +} + +static void submission_tasklet(struct tasklet_struct *t) +{ + struct i915_sched *se = from_tasklet(se, t, tasklet); + struct intel_engine_cs * const engine = + container_of(se, typeof(*engine), sched); + struct i915_request *post[2 * EXECLIST_MAX_PORTS]; + struct i915_request **inactive; + + rcu_read_lock(); + inactive = process_csb(&engine->execlists, post); + GEM_BUG_ON(inactive - post > ARRAY_SIZE(post)); + + if (!i915_sched_is_idle(se)) + dequeue(se, engine); + + post_process_csb(post, inactive); + rcu_read_unlock(); +} + +static void reset_prepare(struct intel_engine_cs *engine) +{ + GEM_TRACE("%s\n", engine->name); + + i915_sched_disable(intel_engine_get_scheduler(engine)); + + intel_ring_submission_reset_prepare(engine); +} + +static inline void clear_ports(struct i915_request **ports, int count) +{ + memset_p((void **)ports, NULL, count); +} + +static struct i915_request ** +cancel_port_requests(struct intel_engine_execlists * const el, + struct i915_request **inactive) +{ + struct i915_request * const *port; + + clear_ports(el->pending, ARRAY_SIZE(el->pending)); + + /* Mark the end of active before we overwrite *active */ + for (port = xchg(&el->active, el->pending); *port; port++) + *inactive++ = *port; + clear_ports(el->inflight, ARRAY_SIZE(el->inflight)); + + smp_wmb(); /* complete the seqlock for execlists_active() */ + WRITE_ONCE(el->active, el->inflight); + + return inactive; +} + +static void __ring_rewind(struct intel_engine_cs *engine, bool stalled) +{ + struct i915_sched *se = intel_engine_get_scheduler(engine); + struct i915_request *rq; + unsigned long flags; + + rcu_read_lock(); + spin_lock_irqsave(&se->lock, flags); + rq = __i915_sched_rewind_requests(engine); + spin_unlock_irqrestore(&se->lock, flags); + if (rq && __i915_request_has_started(rq)) + __i915_request_reset(rq, stalled); + rcu_read_unlock(); +} + +static void ring_reset_csb(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request *post[2 * EXECLIST_MAX_PORTS]; + struct i915_request **inactive; + + rcu_read_lock(); + inactive = cancel_port_requests(el, post); + + /* Clear the global submission state, we will submit from scratch */ + intel_ring_reset(engine->legacy.ring, 0); + set_current_context(&engine->legacy.context, NULL); + + post_process_csb(post, inactive); + rcu_read_unlock(); +} + +static void ring_reset_rewind(struct intel_engine_cs *engine, bool stalled) +{ + ring_reset_csb(engine); + __ring_rewind(engine, stalled); +} + +static void ring_reset_cancel(struct intel_engine_cs *engine) +{ + struct i915_sched *se = intel_engine_get_scheduler(engine); + struct i915_request *rq, *rn; + struct i915_priolist *p; + unsigned long flags; + + ring_reset_csb(engine); + + rcu_read_lock(); + spin_lock_irqsave(&se->lock, flags); + + /* Mark all submitted requests as skipped. */ + list_for_each_entry(rq, &se->requests, sched.link) + i915_request_mark_eio(rq); + intel_engine_signal_breadcrumbs(engine); + + /* Flush the queued requests to the timeline list (for retiring). */ + for_each_priolist(p, &se->queue) { + priolist_for_each_request_safe(rq, rn, p) { + i915_request_mark_eio(rq); + __i915_request_submit(rq); + } + i915_priolist_advance(&se->queue, p); + } + GEM_BUG_ON(!i915_sched_is_idle(se)); + + /* Remaining _unready_ requests will be nop'ed when submitted */ + + spin_unlock_irqrestore(&se->lock, flags); + rcu_read_unlock(); +} + +static void reset_finish(struct intel_engine_cs *engine) +{ + intel_ring_submission_reset_finish(engine); + i915_sched_enable(intel_engine_get_scheduler(engine)); +} + +static void submission_park(struct intel_engine_cs *engine) +{ + /* drain the submit queue */ + intel_breadcrumbs_unpin_irq(engine->breadcrumbs); + intel_engine_kick_scheduler(engine); +} + +static void submission_unpark(struct intel_engine_cs *engine) +{ + intel_breadcrumbs_pin_irq(engine->breadcrumbs); +} + +static void ring_context_destroy(struct kref *ref) +{ + struct intel_context *ce = container_of(ref, typeof(*ce), ref); + + GEM_BUG_ON(intel_context_is_pinned(ce)); + + if (ce->state) + i915_vma_put(ce->state); + if (test_bit(CONTEXT_ALLOC_BIT, &ce->flags)) + intel_ring_put(ce->ring); + + intel_context_fini(ce); + intel_context_free(ce); +} + +static int alloc_context_vma(struct intel_context *ce) + +{ + struct intel_engine_cs *engine = ce->engine; + struct drm_i915_gem_object *obj; + struct i915_vma *vma; + int err; + + obj = i915_gem_object_create_shmem(engine->i915, engine->context_size); + if (IS_ERR(obj)) + return PTR_ERR(obj); + + /* + * Try to make the context utilize L3 as well as LLC. + * + * On VLV we don't have L3 controls in the PTEs so we + * shouldn't touch the cache level, especially as that + * would make the object snooped which might have a + * negative performance impact. + * + * Snooping is required on non-llc platforms in execlist + * mode, but since all GGTT accesses use PAT entry 0 we + * get snooping anyway regardless of cache_level. + * + * This is only applicable for Ivy Bridge devices since + * later platforms don't have L3 control bits in the PTE. + */ + if (IS_IVYBRIDGE(engine->i915)) + i915_gem_object_set_cache_coherency(obj, I915_CACHE_L3_LLC); + + if (engine->default_state) { + void *vaddr; + + vaddr = i915_gem_object_pin_map(obj, I915_MAP_WB); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto err_obj; + } + + shmem_read(engine->default_state, 0, + vaddr, engine->context_size); + __set_bit(CONTEXT_VALID_BIT, &ce->flags); + + i915_gem_object_flush_map(obj); + i915_gem_object_unpin_map(obj); + } + + vma = i915_vma_instance(obj, &engine->gt->ggtt->vm, NULL); + if (IS_ERR(vma)) { + err = PTR_ERR(vma); + goto err_obj; + } + + ce->state = vma; + return 0; + +err_obj: + i915_gem_object_put(obj); + return err; +} + +static struct intel_timeline *pinned_timeline(struct intel_context *ce) +{ + struct intel_timeline *tl = fetch_and_zero(&ce->timeline); + + return intel_timeline_create_from_engine(ce->engine, + page_unmask_bits(tl)); +} + +static int alloc_timeline(struct intel_context *ce) +{ + struct intel_engine_cs *engine = ce->engine; + struct intel_timeline *tl; + + if (unlikely(ce->timeline)) + tl = pinned_timeline(ce); + else + tl = intel_timeline_create(engine->gt); + if (IS_ERR(tl)) + return PTR_ERR(tl); + + ce->timeline = tl; + return 0; +} + +static int ring_context_alloc(struct intel_context *ce) +{ + struct intel_engine_cs *engine = ce->engine; + struct intel_ring *ring; + int err; + + GEM_BUG_ON(ce->state); + if (engine->context_size) { + err = alloc_context_vma(ce); + if (err) + return err; + } + + if (!page_mask_bits(ce->timeline)) { + err = alloc_timeline(ce); + if (err) + goto err_vma; + } + + ring = intel_engine_create_ring(engine, + (unsigned long)ce->ring | + INTEL_RING_CREATE_INTERNAL); + if (IS_ERR(ring)) { + err = PTR_ERR(ring); + goto err_timeline; + } + ce->ring = ring; + + return 0; + +err_timeline: + intel_timeline_put(ce->timeline); +err_vma: + if (ce->state) { + i915_vma_put(ce->state); + ce->state = NULL; + } + return err; +} + +static int ring_context_pre_pin(struct intel_context *ce, + struct i915_gem_ww_ctx *ww, + void **unused) +{ + return 0; +} + +static int ring_context_pin(struct intel_context *ce, void *unused) +{ + return 0; +} + +static void ring_context_unpin(struct intel_context *ce) +{ +} + +static void ring_context_post_unpin(struct intel_context *ce) +{ +} + +static void ring_context_reset(struct intel_context *ce) +{ + intel_ring_reset(ce->ring, 0); + clear_bit(CONTEXT_VALID_BIT, &ce->flags); +} + +static const struct intel_context_ops ring_context_ops = { + .flags = COPS_HAS_INFLIGHT, + + .alloc = ring_context_alloc, + + .pre_pin = ring_context_pre_pin, + .pin = ring_context_pin, + .unpin = ring_context_unpin, + .post_unpin = ring_context_post_unpin, + + .enter = intel_context_enter_engine, + .exit = intel_context_exit_engine, + + .reset = ring_context_reset, + .destroy = ring_context_destroy, +}; + +static int ring_request_alloc(struct i915_request *rq) +{ + int ret; + + GEM_BUG_ON(!intel_context_is_pinned(rq->context)); + + /* + * Flush enough space to reduce the likelihood of waiting after + * we start building the request - in which case we will just + * have to repeat work. + */ + rq->reserved_space += LEGACY_REQUEST_SIZE; + + /* Unconditionally invalidate GPU caches and TLBs. */ + ret = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (ret) + return ret; + + rq->reserved_space -= LEGACY_REQUEST_SIZE; + return 0; +} + +static void set_default_submission(struct intel_engine_cs *engine) +{ + engine->sched.submit_request = i915_request_enqueue; +} + +static void ring_release(struct intel_engine_cs *engine) +{ + intel_engine_cleanup_common(engine); + + set_current_context(&engine->legacy.context, NULL); + + intel_ring_unpin(engine->legacy.ring); + intel_ring_put(engine->legacy.ring); +} + +static void setup_irq(struct intel_engine_cs *engine) +{ +} + +static void setup_common(struct intel_engine_cs *engine) +{ + struct drm_i915_private *i915 = engine->i915; + + /* gen8+ are only supported with execlists */ + GEM_BUG_ON(INTEL_GEN(i915) >= 8); + GEM_BUG_ON(INTEL_GEN(i915) < 8); + + setup_irq(engine); + + engine->park = submission_park; + engine->unpark = submission_unpark; + + engine->resume = intel_ring_submission_resume; + engine->sanitize = intel_ring_submission_sanitize; + + engine->reset.prepare = reset_prepare; + engine->reset.rewind = ring_reset_rewind; + engine->reset.cancel = ring_reset_cancel; + engine->reset.finish = reset_finish; + + engine->cops = &ring_context_ops; + engine->request_alloc = ring_request_alloc; + + engine->set_default_submission = set_default_submission; +} + +static void setup_rcs(struct intel_engine_cs *engine) +{ +} + +static void setup_vcs(struct intel_engine_cs *engine) +{ +} + +static void setup_bcs(struct intel_engine_cs *engine) +{ +} + +static void setup_vecs(struct intel_engine_cs *engine) +{ + GEM_BUG_ON(!IS_HASWELL(engine->i915)); +} + +static unsigned int global_ring_size(void) +{ + /* Enough space to hold 2 clients and the context switch */ + return roundup_pow_of_two(EXECLIST_MAX_PORTS * SZ_16K + SZ_4K); +} + +int intel_ring_scheduler_setup(struct intel_engine_cs *engine) +{ + struct intel_ring *ring; + int err; + + GEM_BUG_ON(HAS_EXECLISTS(engine->i915)); + + tasklet_setup(&engine->sched.tasklet, submission_tasklet); + __set_bit(I915_SCHED_ACTIVE_BIT, &engine->sched.flags); + __set_bit(I915_SCHED_NEEDS_BREADCRUMB_BIT, &engine->sched.flags); + + setup_common(engine); + + switch (engine->class) { + case RENDER_CLASS: + setup_rcs(engine); + break; + case VIDEO_DECODE_CLASS: + setup_vcs(engine); + break; + case COPY_ENGINE_CLASS: + setup_bcs(engine); + break; + case VIDEO_ENHANCEMENT_CLASS: + setup_vecs(engine); + break; + default: + MISSING_CASE(engine->class); + return -ENODEV; + } + + ring = intel_engine_create_ring(engine, global_ring_size()); + if (IS_ERR(ring)) { + err = PTR_ERR(ring); + goto err; + } + + err = intel_ring_pin(ring, NULL); + if (err) + goto err_ring; + + GEM_BUG_ON(engine->legacy.ring); + engine->legacy.ring = ring; + + engine->flags |= I915_ENGINE_SUPPORTS_STATS; + + /* Finally, take ownership and responsibility for cleanup! */ + engine->release = ring_release; + return 0; + +err_ring: + intel_ring_put(ring); +err: + intel_engine_cleanup_common(engine); + return err; +} diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c index ef80f47f468a..ede148c7b2bd 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c @@ -14,6 +14,7 @@ #include "intel_gt.h" #include "intel_reset.h" #include "intel_ring.h" +#include "intel_ring_submission.h" #include "shmem_utils.h" /* Rough estimate of the typical request size, performing a flush, @@ -176,7 +177,7 @@ static bool stop_ring(struct intel_engine_cs *engine) return (ENGINE_READ_FW(engine, RING_HEAD) & HEAD_ADDR) == 0; } -static int xcs_resume(struct intel_engine_cs *engine) +int intel_ring_submission_resume(struct intel_engine_cs *engine) { struct intel_ring *ring = engine->legacy.ring; @@ -264,7 +265,7 @@ static void sanitize_hwsp(struct intel_engine_cs *engine) intel_timeline_reset_seqno(tl); } -static void xcs_sanitize(struct intel_engine_cs *engine) +void intel_ring_submission_sanitize(struct intel_engine_cs *engine) { /* * Poison residual state on resume, in case the suspend didn't! @@ -289,7 +290,7 @@ static void xcs_sanitize(struct intel_engine_cs *engine) clflush_cache_range(engine->status_page.addr, PAGE_SIZE); } -static void reset_prepare(struct intel_engine_cs *engine) +void intel_ring_submission_reset_prepare(struct intel_engine_cs *engine) { /* * We stop engines, otherwise we might get failed reset and a @@ -388,7 +389,7 @@ static void reset_rewind(struct intel_engine_cs *engine, bool stalled) spin_unlock_irqrestore(&se->lock, flags); } -static void reset_finish(struct intel_engine_cs *engine) +void intel_ring_submission_reset_finish(struct intel_engine_cs *engine) { } @@ -1027,13 +1028,13 @@ static void setup_common(struct intel_engine_cs *engine) setup_irq(engine); - engine->resume = xcs_resume; - engine->sanitize = xcs_sanitize; + engine->resume = intel_ring_submission_resume; + engine->sanitize = intel_ring_submission_sanitize; - engine->reset.prepare = reset_prepare; + engine->reset.prepare = intel_ring_submission_reset_prepare; engine->reset.rewind = reset_rewind; engine->reset.cancel = reset_cancel; - engine->reset.finish = reset_finish; + engine->reset.finish = intel_ring_submission_reset_finish; engine->cops = &ring_context_ops; engine->request_alloc = ring_request_alloc; diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.h b/drivers/gpu/drm/i915/gt/intel_ring_submission.h new file mode 100644 index 000000000000..59a43c221748 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_RING_SUBMISSION_H__ +#define __INTEL_RING_SUBMISSION_H__ + +struct intel_engine_cs; + +void intel_ring_submission_reset_prepare(struct intel_engine_cs *engine); +void intel_ring_submission_reset_finish(struct intel_engine_cs *engine); + +int intel_ring_submission_resume(struct intel_engine_cs *engine); +void intel_ring_submission_sanitize(struct intel_engine_cs *engine); + +#endif /* __INTEL_RING_SUBMISSION_H__ */