From patchwork Fri Jun 5 12:23:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589563 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B175912 for ; Fri, 5 Jun 2020 12:23:54 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 236CA206E6 for ; Fri, 5 Jun 2020 12:23:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 236CA206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C92736E144; Fri, 5 Jun 2020 12:23:48 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 26FA66E144 for ; Fri, 5 Jun 2020 12:23:46 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406189-1500050 for multiple; Fri, 05 Jun 2020 13:23:37 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:25 +0100 Message-Id: <20200605122334.2798-1-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 01/10] drm/i915/gt: Set timeslicing priority from queue X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If we only submit the first port, leaving the second empty yet have ready requests pending in the queue, use that to set the timeslicing priority (i.e. the priority at which we will decided to enabling timeslicing and evict the currently active context if the queue is of equal priority after its quantum expired). Signed-off-by: Chris Wilson Reviewed-by: Mika Kuoppala --- drivers/gpu/drm/i915/gt/intel_lrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/i915/gt/intel_lrc.c b/drivers/gpu/drm/i915/gt/intel_lrc.c index 33b7173b7195..92c3368ffcbd 100644 --- a/drivers/gpu/drm/i915/gt/intel_lrc.c +++ b/drivers/gpu/drm/i915/gt/intel_lrc.c @@ -1966,7 +1966,7 @@ static int switch_prio(struct intel_engine_cs *engine, const struct i915_request *rq) { if (list_is_last(&rq->sched.link, &engine->active.requests)) - return INT_MIN; + return engine->execlists.queue_priority_hint; return rq_prio(list_next_entry(rq, sched.link)); } From patchwork Fri Jun 5 12:23:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589567 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 789BF90 for ; Fri, 5 Jun 2020 12:23:56 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 60FC0206E6 for ; Fri, 5 Jun 2020 12:23:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 60FC0206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2E4B66E85D; Fri, 5 Jun 2020 12:23:49 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 202516E119 for ; Fri, 5 Jun 2020 12:23:46 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406190-1500050 for multiple; Fri, 05 Jun 2020 13:23:37 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:26 +0100 Message-Id: <20200605122334.2798-2-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 02/10] drm/i915/gt: Always check to enable timeslicing if not submitting X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" We may choose not to submit for a number of reasons, yet not fill both ELSP. In which case we must start timeslicing (there will be no ACK event on which to hook the start) if the queue would benefit from the currently active context being evicted. Signed-off-by: Chris Wilson Reviewed-by: Mika Kuoppala --- drivers/gpu/drm/i915/gt/intel_lrc.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_lrc.c b/drivers/gpu/drm/i915/gt/intel_lrc.c index 92c3368ffcbd..d55a5e0466e5 100644 --- a/drivers/gpu/drm/i915/gt/intel_lrc.c +++ b/drivers/gpu/drm/i915/gt/intel_lrc.c @@ -2362,10 +2362,8 @@ static void execlists_dequeue(struct intel_engine_cs *engine) if (last->context == rq->context) goto done; - if (i915_request_has_sentinel(last)) { - start_timeslice(engine, rq_prio(rq)); + if (i915_request_has_sentinel(last)) goto done; - } /* * If GVT overrides us we only ever submit @@ -2446,6 +2444,7 @@ static void execlists_dequeue(struct intel_engine_cs *engine) set_preempt_timeout(engine, *active); execlists_submit_ports(engine); } else { + start_timeslice(engine, execlists->queue_priority_hint); skip_submit: ring_set_paused(engine, 0); } From patchwork Fri Jun 5 12:23:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589571 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18E30912 for ; Fri, 5 Jun 2020 12:23:58 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 019E7206E6 for ; Fri, 5 Jun 2020 12:23:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 019E7206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 99B7A6E8A0; Fri, 5 Jun 2020 12:23:52 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 18AE96E859 for ; Fri, 5 Jun 2020 12:23:47 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406191-1500050 for multiple; Fri, 05 Jun 2020 13:23:38 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:27 +0100 Message-Id: <20200605122334.2798-3-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 03/10] Restore "drm/i915: drop engine_pin/unpin_breadcrumbs_irq" X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" This was removed in commit 478ffad6d690 ("drm/i915: drop engine_pin/unpin_breadcrumbs_irq") as the last user had been removed, but now there is a promise of a new user in the next patch. Signed-off-by: Chris Wilson Reviewed-by: Mika Kuoppala --- drivers/gpu/drm/i915/gt/intel_breadcrumbs.c | 22 +++++++++++++++++++++ drivers/gpu/drm/i915/gt/intel_engine.h | 3 +++ 2 files changed, 25 insertions(+) diff --git a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c index d907d538176e..03c14ab86d95 100644 --- a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c +++ b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c @@ -220,6 +220,28 @@ static void signal_irq_work(struct irq_work *work) } } +void intel_engine_pin_breadcrumbs_irq(struct intel_engine_cs *engine) +{ + struct intel_breadcrumbs *b = &engine->breadcrumbs; + + spin_lock_irq(&b->irq_lock); + if (!b->irq_enabled++) + irq_enable(engine); + GEM_BUG_ON(!b->irq_enabled); /* no overflow! */ + spin_unlock_irq(&b->irq_lock); +} + +void intel_engine_unpin_breadcrumbs_irq(struct intel_engine_cs *engine) +{ + struct intel_breadcrumbs *b = &engine->breadcrumbs; + + spin_lock_irq(&b->irq_lock); + GEM_BUG_ON(!b->irq_enabled); /* no underflow! */ + if (!--b->irq_enabled) + irq_disable(engine); + spin_unlock_irq(&b->irq_lock); +} + static bool __intel_breadcrumbs_arm_irq(struct intel_breadcrumbs *b) { struct intel_engine_cs *engine = diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 791897f8d847..043462b6ce1f 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -226,6 +226,9 @@ void intel_engine_init_execlists(struct intel_engine_cs *engine); void intel_engine_init_breadcrumbs(struct intel_engine_cs *engine); void intel_engine_fini_breadcrumbs(struct intel_engine_cs *engine); +void intel_engine_pin_breadcrumbs_irq(struct intel_engine_cs *engine); +void intel_engine_unpin_breadcrumbs_irq(struct intel_engine_cs *engine); + void intel_engine_disarm_breadcrumbs(struct intel_engine_cs *engine); static inline void From patchwork Fri Jun 5 12:23:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589565 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65EA3912 for ; Fri, 5 Jun 2020 12:23:55 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4E536206E6 for ; Fri, 5 Jun 2020 12:23:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E536206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1EE646E859; Fri, 5 Jun 2020 12:23:49 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 372666E859 for ; Fri, 5 Jun 2020 12:23:46 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406192-1500050 for multiple; Fri, 05 Jun 2020 13:23:38 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:28 +0100 Message-Id: <20200605122334.2798-4-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 04/10] drm/i915/gt: Couple tasklet scheduling for all CS interrupts X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If any engine asks for the tasklet to be kicked from the CS interrupt, do so. Signed-off-by: Chris Wilson Reviewed-by: Mika Kuoppala --- drivers/gpu/drm/i915/gt/intel_gt_irq.c | 17 ++++++++++++----- drivers/gpu/drm/i915/gt/intel_gt_irq.h | 3 +++ drivers/gpu/drm/i915/gt/intel_rps.c | 2 +- drivers/gpu/drm/i915/i915_irq.c | 8 ++++---- 4 files changed, 20 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_gt_irq.c b/drivers/gpu/drm/i915/gt/intel_gt_irq.c index 0cc7dd54f4f9..28edf314a319 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_irq.c +++ b/drivers/gpu/drm/i915/gt/intel_gt_irq.c @@ -60,6 +60,13 @@ cs_irq_handler(struct intel_engine_cs *engine, u32 iir) tasklet_hi_schedule(&engine->execlists.tasklet); } +void gen2_engine_cs_irq(struct intel_engine_cs *engine) +{ + intel_engine_signal_breadcrumbs(engine); + if (intel_engine_needs_breadcrumb_tasklet(engine)) + tasklet_hi_schedule(&engine->execlists.tasklet); +} + static u32 gen11_gt_engine_identity(struct intel_gt *gt, const unsigned int bank, const unsigned int bit) @@ -273,9 +280,9 @@ void gen11_gt_irq_postinstall(struct intel_gt *gt) void gen5_gt_irq_handler(struct intel_gt *gt, u32 gt_iir) { if (gt_iir & GT_RENDER_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine_class[RENDER_CLASS][0]); + gen2_engine_cs_irq(gt->engine_class[RENDER_CLASS][0]); if (gt_iir & ILK_BSD_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine_class[VIDEO_DECODE_CLASS][0]); + gen2_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][0]); } static void gen7_parity_error_irq_handler(struct intel_gt *gt, u32 iir) @@ -299,11 +306,11 @@ static void gen7_parity_error_irq_handler(struct intel_gt *gt, u32 iir) void gen6_gt_irq_handler(struct intel_gt *gt, u32 gt_iir) { if (gt_iir & GT_RENDER_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine_class[RENDER_CLASS][0]); + gen2_engine_cs_irq(gt->engine_class[RENDER_CLASS][0]); if (gt_iir & GT_BSD_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine_class[VIDEO_DECODE_CLASS][0]); + gen2_engine_cs_irq(gt->engine_class[VIDEO_DECODE_CLASS][0]); if (gt_iir & GT_BLT_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine_class[COPY_ENGINE_CLASS][0]); + gen2_engine_cs_irq(gt->engine_class[COPY_ENGINE_CLASS][0]); if (gt_iir & (GT_BLT_CS_ERROR_INTERRUPT | GT_BSD_CS_ERROR_INTERRUPT | diff --git a/drivers/gpu/drm/i915/gt/intel_gt_irq.h b/drivers/gpu/drm/i915/gt/intel_gt_irq.h index 886c5cf408a2..6c69cd563fe1 100644 --- a/drivers/gpu/drm/i915/gt/intel_gt_irq.h +++ b/drivers/gpu/drm/i915/gt/intel_gt_irq.h @@ -9,6 +9,7 @@ #include +struct intel_engine_cs; struct intel_gt; #define GEN8_GT_IRQS (GEN8_GT_RCS_IRQ | \ @@ -19,6 +20,8 @@ struct intel_gt; GEN8_GT_PM_IRQ | \ GEN8_GT_GUC_IRQ) +void gen2_engine_cs_irq(struct intel_engine_cs *engine); + void gen11_gt_irq_reset(struct intel_gt *gt); void gen11_gt_irq_postinstall(struct intel_gt *gt); void gen11_gt_irq_handler(struct intel_gt *gt, const u32 master_ctl); diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c index 2f59fc6df3c2..2e4ddc9ca09d 100644 --- a/drivers/gpu/drm/i915/gt/intel_rps.c +++ b/drivers/gpu/drm/i915/gt/intel_rps.c @@ -1741,7 +1741,7 @@ void gen6_rps_irq_handler(struct intel_rps *rps, u32 pm_iir) return; if (pm_iir & PM_VEBOX_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(gt->engine[VECS0]); + gen2_engine_cs_irq(gt->engine[VECS0]); if (pm_iir & PM_VEBOX_CS_ERROR_INTERRUPT) DRM_DEBUG("Command parser error, pm_iir 0x%08x\n", pm_iir); diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c index 8e823ba25f5f..b64f3b3bca70 100644 --- a/drivers/gpu/drm/i915/i915_irq.c +++ b/drivers/gpu/drm/i915/i915_irq.c @@ -3686,7 +3686,7 @@ static irqreturn_t i8xx_irq_handler(int irq, void *arg) intel_uncore_write16(&dev_priv->uncore, GEN2_IIR, iir); if (iir & I915_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]); + gen2_engine_cs_irq(dev_priv->gt.engine[RCS0]); if (iir & I915_MASTER_ERROR_INTERRUPT) i8xx_error_irq_handler(dev_priv, eir, eir_stuck); @@ -3791,7 +3791,7 @@ static irqreturn_t i915_irq_handler(int irq, void *arg) I915_WRITE(GEN2_IIR, iir); if (iir & I915_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]); + gen2_engine_cs_irq(dev_priv->gt.engine[RCS0]); if (iir & I915_MASTER_ERROR_INTERRUPT) i9xx_error_irq_handler(dev_priv, eir, eir_stuck); @@ -3933,10 +3933,10 @@ static irqreturn_t i965_irq_handler(int irq, void *arg) I915_WRITE(GEN2_IIR, iir); if (iir & I915_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(dev_priv->gt.engine[RCS0]); + gen2_engine_cs_irq(dev_priv->gt.engine[RCS0]); if (iir & I915_BSD_USER_INTERRUPT) - intel_engine_signal_breadcrumbs(dev_priv->gt.engine[VCS0]); + gen2_engine_cs_irq(dev_priv->gt.engine[VCS0]); if (iir & I915_MASTER_ERROR_INTERRUPT) i9xx_error_irq_handler(dev_priv, eir, eir_stuck); From patchwork Fri Jun 5 12:23:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589561 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E38D990 for ; Fri, 5 Jun 2020 12:23:51 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBBF9206E6 for ; Fri, 5 Jun 2020 12:23:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBBF9206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BB0976E119; Fri, 5 Jun 2020 12:23:48 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2E2036E849 for ; Fri, 5 Jun 2020 12:23:46 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406193-1500050 for multiple; Fri, 05 Jun 2020 13:23:38 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:29 +0100 Message-Id: <20200605122334.2798-5-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 05/10] drm/i915/gt: Support creation of 'internal' rings X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To support legacy ring buffer scheduling, we want a virtual ringbuffer for each client. These rings are purely for holding the requests as they are being constructed on the CPU and never accessed by the GPU, so they should not be bound into the GGTT, and we can use plain old WB mapped pages. As they are not bound, we need to nerf a few assumptions that a rq->ring is in the GGTT. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/gt/intel_context.c | 2 +- drivers/gpu/drm/i915/gt/intel_engine_cs.c | 17 ++---- drivers/gpu/drm/i915/gt/intel_ring.c | 63 ++++++++++++++-------- drivers/gpu/drm/i915/gt/intel_ring.h | 12 ++++- drivers/gpu/drm/i915/gt/intel_ring_types.h | 2 + 5 files changed, 57 insertions(+), 39 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_context.c b/drivers/gpu/drm/i915/gt/intel_context.c index e4aece20bc80..fd71977c010a 100644 --- a/drivers/gpu/drm/i915/gt/intel_context.c +++ b/drivers/gpu/drm/i915/gt/intel_context.c @@ -127,7 +127,7 @@ int __intel_context_do_pin(struct intel_context *ce) goto err_active; CE_TRACE(ce, "pin ring:{start:%08x, head:%04x, tail:%04x}\n", - i915_ggtt_offset(ce->ring->vma), + intel_ring_address(ce->ring), ce->ring->head, ce->ring->tail); smp_mb__before_atomic(); /* flush pin before it is visible */ diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c index e37490d459c2..4b36378af119 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c @@ -1259,7 +1259,7 @@ static int print_ring(char *buf, int sz, struct i915_request *rq) len = scnprintf(buf, sz, "ring:{start:%08x, hwsp:%08x, seqno:%08x, runtime:%llums}, ", - i915_ggtt_offset(rq->ring->vma), + intel_ring_address(rq->ring), tl ? tl->hwsp_offset : 0, hwsp_seqno(rq), DIV_ROUND_CLOSEST_ULL(intel_context_get_total_runtime_ns(rq->context), @@ -1540,7 +1540,7 @@ void intel_engine_dump(struct intel_engine_cs *engine, print_request(m, rq, "\t\tactive "); drm_printf(m, "\t\tring->start: 0x%08x\n", - i915_ggtt_offset(rq->ring->vma)); + intel_ring_address(rq->ring)); drm_printf(m, "\t\tring->head: 0x%08x\n", rq->ring->head); drm_printf(m, "\t\tring->tail: 0x%08x\n", @@ -1619,13 +1619,6 @@ ktime_t intel_engine_get_busy_time(struct intel_engine_cs *engine) return total; } -static bool match_ring(struct i915_request *rq) -{ - u32 ring = ENGINE_READ(rq->engine, RING_START); - - return ring == i915_ggtt_offset(rq->ring->vma); -} - struct i915_request * intel_engine_find_active_request(struct intel_engine_cs *engine) { @@ -1665,11 +1658,7 @@ intel_engine_find_active_request(struct intel_engine_cs *engine) continue; if (!i915_request_started(request)) - continue; - - /* More than one preemptible request may match! */ - if (!match_ring(request)) - continue; + break; active = request; break; diff --git a/drivers/gpu/drm/i915/gt/intel_ring.c b/drivers/gpu/drm/i915/gt/intel_ring.c index 8cda1b7e17ba..438637996ab5 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.c +++ b/drivers/gpu/drm/i915/gt/intel_ring.c @@ -24,33 +24,42 @@ unsigned int intel_ring_update_space(struct intel_ring *ring) int intel_ring_pin(struct intel_ring *ring) { struct i915_vma *vma = ring->vma; - unsigned int flags; void *addr; int ret; if (atomic_fetch_inc(&ring->pin_count)) return 0; - /* Ring wraparound at offset 0 sometimes hangs. No idea why. */ - flags = PIN_OFFSET_BIAS | i915_ggtt_pin_bias(vma); + if (!(ring->flags & INTEL_RING_CREATE_INTERNAL)) { + unsigned int pin; - if (vma->obj->stolen) - flags |= PIN_MAPPABLE; - else - flags |= PIN_HIGH; + /* Ring wraparound at offset 0 sometimes hangs. No idea why. */ + pin |= PIN_OFFSET_BIAS | i915_ggtt_pin_bias(vma); - ret = i915_ggtt_pin(vma, 0, flags); - if (unlikely(ret)) - goto err_unpin; + if (vma->obj->stolen) + pin |= PIN_MAPPABLE; + else + pin |= PIN_HIGH; - if (i915_vma_is_map_and_fenceable(vma)) - addr = (void __force *)i915_vma_pin_iomap(vma); - else - addr = i915_gem_object_pin_map(vma->obj, - i915_coherent_map_type(vma->vm->i915)); - if (IS_ERR(addr)) { - ret = PTR_ERR(addr); - goto err_ring; + ret = i915_ggtt_pin(vma, 0, pin); + if (unlikely(ret)) + goto err_unpin; + + if (i915_vma_is_map_and_fenceable(vma)) + addr = (void __force *)i915_vma_pin_iomap(vma); + else + addr = i915_gem_object_pin_map(vma->obj, + i915_coherent_map_type(vma->vm->i915)); + if (IS_ERR(addr)) { + ret = PTR_ERR(addr); + goto err_ring; + } + } else { + addr = i915_gem_object_pin_map(vma->obj, I915_MAP_WB); + if (IS_ERR(addr)) { + ret = PTR_ERR(addr); + goto err_ring; + } } i915_vma_make_unshrinkable(vma); @@ -91,10 +100,12 @@ void intel_ring_unpin(struct intel_ring *ring) i915_gem_object_unpin_map(vma->obj); i915_vma_make_purgeable(vma); - i915_vma_unpin(vma); + if (!(ring->flags & INTEL_RING_CREATE_INTERNAL)) + i915_vma_unpin(vma); } -static struct i915_vma *create_ring_vma(struct i915_ggtt *ggtt, int size) +static struct i915_vma * +create_ring_vma(struct i915_ggtt *ggtt, int size, unsigned int flags) { struct i915_address_space *vm = &ggtt->vm; struct drm_i915_private *i915 = vm->i915; @@ -102,7 +113,8 @@ static struct i915_vma *create_ring_vma(struct i915_ggtt *ggtt, int size) struct i915_vma *vma; obj = ERR_PTR(-ENODEV); - if (i915_ggtt_has_aperture(ggtt)) + if (!(flags & INTEL_RING_CREATE_INTERNAL) && + i915_ggtt_has_aperture(ggtt)) obj = i915_gem_object_create_stolen(i915, size); if (IS_ERR(obj)) obj = i915_gem_object_create_internal(i915, size); @@ -128,12 +140,14 @@ static struct i915_vma *create_ring_vma(struct i915_ggtt *ggtt, int size) } struct intel_ring * -intel_engine_create_ring(struct intel_engine_cs *engine, int size) +intel_engine_create_ring(struct intel_engine_cs *engine, unsigned int size) { struct drm_i915_private *i915 = engine->i915; + unsigned int flags = size & GENMASK(11, 0); struct intel_ring *ring; struct i915_vma *vma; + size ^= flags; GEM_BUG_ON(!is_power_of_2(size)); GEM_BUG_ON(RING_CTL_SIZE(size) & ~RING_NR_PAGES); @@ -142,8 +156,10 @@ intel_engine_create_ring(struct intel_engine_cs *engine, int size) return ERR_PTR(-ENOMEM); kref_init(&ring->ref); + ring->size = size; ring->wrap = BITS_PER_TYPE(ring->size) - ilog2(size); + ring->flags = flags; /* * Workaround an erratum on the i830 which causes a hang if @@ -156,11 +172,12 @@ intel_engine_create_ring(struct intel_engine_cs *engine, int size) intel_ring_update_space(ring); - vma = create_ring_vma(engine->gt->ggtt, size); + vma = create_ring_vma(engine->gt->ggtt, size, flags); if (IS_ERR(vma)) { kfree(ring); return ERR_CAST(vma); } + ring->vma = vma; return ring; diff --git a/drivers/gpu/drm/i915/gt/intel_ring.h b/drivers/gpu/drm/i915/gt/intel_ring.h index cc0ebca65167..d022fa209325 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring.h +++ b/drivers/gpu/drm/i915/gt/intel_ring.h @@ -9,12 +9,14 @@ #include "i915_gem.h" /* GEM_BUG_ON */ #include "i915_request.h" +#include "i915_vma.h" #include "intel_ring_types.h" struct intel_engine_cs; struct intel_ring * -intel_engine_create_ring(struct intel_engine_cs *engine, int size); +intel_engine_create_ring(struct intel_engine_cs *engine, unsigned int size); +#define INTEL_RING_CREATE_INTERNAL BIT(0) u32 *intel_ring_begin(struct i915_request *rq, unsigned int num_dwords); int intel_ring_cacheline_align(struct i915_request *rq); @@ -137,4 +139,12 @@ __intel_ring_space(unsigned int head, unsigned int tail, unsigned int size) return (head - tail - CACHELINE_BYTES) & (size - 1); } +static inline u32 intel_ring_address(const struct intel_ring *ring) +{ + if (ring->flags & INTEL_RING_CREATE_INTERNAL) + return -1; + + return i915_ggtt_offset(ring->vma); +} + #endif /* INTEL_RING_H */ diff --git a/drivers/gpu/drm/i915/gt/intel_ring_types.h b/drivers/gpu/drm/i915/gt/intel_ring_types.h index 1a189ea00fd8..d927deafcb33 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_types.h +++ b/drivers/gpu/drm/i915/gt/intel_ring_types.h @@ -47,6 +47,8 @@ struct intel_ring { u32 size; u32 wrap; u32 effective_size; + + unsigned long flags; }; #endif /* INTEL_RING_TYPES_H */ From patchwork Fri Jun 5 12:23:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589577 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18A5490 for ; Fri, 5 Jun 2020 12:24:00 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 019AD206E6 for ; Fri, 5 Jun 2020 12:24:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 019AD206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D02256E8CE; Fri, 5 Jun 2020 12:23:55 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 11BFA6E849 for ; Fri, 5 Jun 2020 12:23:47 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406194-1500050 for multiple; Fri, 05 Jun 2020 13:23:38 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:30 +0100 Message-Id: <20200605122334.2798-6-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 06/10] drm/i915/gt: Use client timeline address for seqno writes X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" If we allow for per-client timelines, even with legacy ring submission, we open the door to a world full of possiblities [scheduling and semaphores]. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/gt/gen6_engine_cs.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/gen6_engine_cs.c b/drivers/gpu/drm/i915/gt/gen6_engine_cs.c index ce38d1bcaba3..fa11174bb13b 100644 --- a/drivers/gpu/drm/i915/gt/gen6_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/gen6_engine_cs.c @@ -373,11 +373,10 @@ u32 *gen7_emit_breadcrumb_rcs(struct i915_request *rq, u32 *cs) u32 *gen6_emit_breadcrumb_xcs(struct i915_request *rq, u32 *cs) { - GEM_BUG_ON(i915_request_active_timeline(rq)->hwsp_ggtt != rq->engine->status_page.vma); - GEM_BUG_ON(offset_in_page(i915_request_active_timeline(rq)->hwsp_offset) != I915_GEM_HWS_SEQNO_ADDR); + u32 addr = i915_request_active_timeline(rq)->hwsp_offset; - *cs++ = MI_FLUSH_DW | MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_DW_STORE_INDEX; - *cs++ = I915_GEM_HWS_SEQNO_ADDR | MI_FLUSH_DW_USE_GTT; + *cs++ = MI_FLUSH_DW | MI_FLUSH_DW_OP_STOREDW; + *cs++ = addr | MI_FLUSH_DW_USE_GTT; *cs++ = rq->fence.seqno; *cs++ = MI_USER_INTERRUPT; @@ -391,19 +390,17 @@ u32 *gen6_emit_breadcrumb_xcs(struct i915_request *rq, u32 *cs) #define GEN7_XCS_WA 32 u32 *gen7_emit_breadcrumb_xcs(struct i915_request *rq, u32 *cs) { + u32 addr = i915_request_active_timeline(rq)->hwsp_offset; int i; - GEM_BUG_ON(i915_request_active_timeline(rq)->hwsp_ggtt != rq->engine->status_page.vma); - GEM_BUG_ON(offset_in_page(i915_request_active_timeline(rq)->hwsp_offset) != I915_GEM_HWS_SEQNO_ADDR); - - *cs++ = MI_FLUSH_DW | MI_INVALIDATE_TLB | - MI_FLUSH_DW_OP_STOREDW | MI_FLUSH_DW_STORE_INDEX; - *cs++ = I915_GEM_HWS_SEQNO_ADDR | MI_FLUSH_DW_USE_GTT; + *cs++ = MI_FLUSH_DW | MI_FLUSH_DW_OP_STOREDW; + *cs++ = addr | MI_FLUSH_DW_USE_GTT; *cs++ = rq->fence.seqno; for (i = 0; i < GEN7_XCS_WA; i++) { - *cs++ = MI_STORE_DWORD_INDEX; - *cs++ = I915_GEM_HWS_SEQNO_ADDR; + *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT; + *cs++ = 0; + *cs++ = addr; *cs++ = rq->fence.seqno; } From patchwork Fri Jun 5 12:23:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589575 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C0791667 for ; Fri, 5 Jun 2020 12:23:59 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 54B47206E6 for ; Fri, 5 Jun 2020 12:23:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 54B47206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 71C646E8B7; Fri, 5 Jun 2020 12:23:53 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0747C6E144 for ; Fri, 5 Jun 2020 12:23:47 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406195-1500050 for multiple; Fri, 05 Jun 2020 13:23:38 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:31 +0100 Message-Id: <20200605122334.2798-7-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 07/10] drm/i915/gt: Infrastructure for ring scheduling X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Build a bare bones scheduler to sit on top the global legacy ringbuffer submission. This virtual execlists scheme should be applicable to all older platforms. A key problem we have with the legacy ring buffer submission is that it only allows for FIFO queuing. All clients share the global request queue and must contend for its lock when submitting. As any client may need to wait for external events, all clients must then wait. However, if we stage each client into their own virtual ringbuffer with their own timelines, we can copy the client requests into the global ringbuffer only when they are ready, reordering the submission around stalls. Furthermore, the ability to reorder gives us rudimentarily priority sorting -- although without preemption support, once something is on the GPU it stays on the GPU, and so it is still possible for a hog to delay a high priority request (such as updating the display). However, it does means that in keeping a short submission queue, the high priority request will be next. This design resembles the old guc submission scheduler, for reordering requests onto a global workqueue. The implementation uses the MI_USER_INTERRUPT at the end of every request to track completion, so is more interrupt happy than execlists [which has an interrupt for each context event, albeit two]. Our interrupts on these system are relatively heavy, and in the past we have been able to completely starve Sandybrige by the interrupt traffic. Our interrupt handlers are being much better (in part offloading the work to bottom halves leaving the interrupt itself only dealing with acking the registers) but we can still see the impact of starvation in the uneven submission latency on a saturated system. Overall though, the short sumission queues and extra interrupts do not appear to be affecting throughput (+-10%, some tasks even improve to the reduced request overheads) and improve latency. [Which is a massive improvement since the introduction of Sandybridge!] Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/gt/intel_engine.h | 1 + drivers/gpu/drm/i915/gt/intel_engine_types.h | 1 + .../gpu/drm/i915/gt/intel_ring_scheduler.c | 760 ++++++++++++++++++ .../gpu/drm/i915/gt/intel_ring_submission.c | 13 +- .../gpu/drm/i915/gt/intel_ring_submission.h | 16 + 6 files changed, 786 insertions(+), 6 deletions(-) create mode 100644 drivers/gpu/drm/i915/gt/intel_ring_scheduler.c create mode 100644 drivers/gpu/drm/i915/gt/intel_ring_submission.h diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile index 41a27fd5dbc7..6d98a74da41e 100644 --- a/drivers/gpu/drm/i915/Makefile +++ b/drivers/gpu/drm/i915/Makefile @@ -109,6 +109,7 @@ gt-y += \ gt/intel_renderstate.o \ gt/intel_reset.o \ gt/intel_ring.o \ + gt/intel_ring_scheduler.o \ gt/intel_ring_submission.o \ gt/intel_rps.o \ gt/intel_sseu.o \ diff --git a/drivers/gpu/drm/i915/gt/intel_engine.h b/drivers/gpu/drm/i915/gt/intel_engine.h index 043462b6ce1f..08176117757e 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine.h +++ b/drivers/gpu/drm/i915/gt/intel_engine.h @@ -209,6 +209,7 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine); int intel_engine_resume(struct intel_engine_cs *engine); int intel_ring_submission_setup(struct intel_engine_cs *engine); +int intel_ring_scheduler_setup(struct intel_engine_cs *engine); int intel_engine_stop_cs(struct intel_engine_cs *engine); void intel_engine_cancel_stop_cs(struct intel_engine_cs *engine); diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h index 073c3769e8cc..5d72690fd83e 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h @@ -359,6 +359,7 @@ struct intel_engine_cs { struct { struct intel_ring *ring; struct intel_timeline *timeline; + struct intel_context *context; } legacy; /* diff --git a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c new file mode 100644 index 000000000000..c8cd435d1c51 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c @@ -0,0 +1,760 @@ +// SPDX-License-Identifier: MIT +/* + * Copyright © 2020 Intel Corporation + */ + +#include + +#include + +#include "i915_drv.h" +#include "intel_context.h" +#include "intel_gt.h" +#include "intel_gt_pm.h" +#include "intel_gt_requests.h" +#include "intel_reset.h" +#include "intel_ring.h" +#include "intel_ring_submission.h" +#include "shmem_utils.h" + +/* + * Rough estimate of the typical request size, performing a flush, + * set-context and then emitting the batch. + */ +#define LEGACY_REQUEST_SIZE 200 + +static inline int rq_prio(const struct i915_request *rq) +{ + return rq->sched.attr.priority; +} + +static inline struct i915_priolist *to_priolist(struct rb_node *rb) +{ + return rb_entry(rb, struct i915_priolist, node); +} + +static inline int queue_prio(struct rb_node *rb) +{ + return rb ? to_priolist(rb)->priority : INT_MIN; +} + +static inline bool reset_in_progress(const struct intel_engine_execlists *el) +{ + return unlikely(!__tasklet_is_enabled(&el->tasklet)); +} + +static void +set_current_context(struct intel_context **ptr, struct intel_context *ce) +{ + if (ce) + intel_context_get(ce); + + ce = xchg(ptr, ce); + + if (ce) + intel_context_put(ce); +} + +static struct i915_request * +schedule_in(struct intel_engine_cs *engine, struct i915_request *rq) +{ + __intel_gt_pm_get(engine->gt); + return i915_request_get(rq); +} + +static void +schedule_out(struct intel_engine_cs *engine, struct i915_request *rq) +{ + struct intel_context *ce = rq->context; + + if (list_is_last_rcu(&rq->link, &ce->timeline->requests)) + intel_engine_add_retire(engine, ce->timeline); + + i915_request_put(rq); + intel_gt_pm_put_async(engine->gt); +} + +static void reset_prepare(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + unsigned long flags; + + GEM_TRACE("%s\n", engine->name); + + __tasklet_disable_sync_once(&el->tasklet); + GEM_BUG_ON(!reset_in_progress(el)); + + /* And flush any current direct submission. */ + spin_lock_irqsave(&engine->active.lock, flags); + spin_unlock_irqrestore(&engine->active.lock, flags); + + intel_ring_submission_reset_prepare(engine); +} + +static void reset_queue_priority(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + + el->queue_priority_hint = queue_prio(rb_first_cached(&el->queue)); +} + +static struct i915_request * +__unwind_incomplete_requests(struct intel_engine_cs *engine) +{ + struct i915_request *rq, *rn, *active = NULL; + struct list_head *uninitialized_var(pl); + int prio = I915_PRIORITY_INVALID; + + lockdep_assert_held(&engine->active.lock); + + list_for_each_entry_safe_reverse(rq, rn, + &engine->active.requests, + sched.link) { + if (i915_request_completed(rq)) + break; + + __i915_request_unsubmit(rq); + + GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID); + if (rq_prio(rq) != prio) { + prio = rq_prio(rq); + pl = i915_sched_lookup_priolist(engine, prio); + } + GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root)); + + list_move(&rq->sched.link, pl); + set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags); + + active = rq; + } + + reset_queue_priority(engine); + + return active; +} + +static inline void clear_ports(struct i915_request **ports, int count) +{ + memset_p((void **)ports, NULL, count); +} + +static void cancel_port_requests(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request * const *port; + + clear_ports(el->pending, ARRAY_SIZE(el->pending)); + for (port = xchg(&el->active, el->pending); *port; port++) + schedule_out(engine, *port); + clear_ports(el->inflight, ARRAY_SIZE(el->inflight)); + + smp_wmb(); /* complete the seqlock for execlists_active() */ + WRITE_ONCE(el->active, el->inflight); +} + +static void __ring_rewind(struct intel_engine_cs *engine, bool stalled) +{ + struct i915_request *rq; + + rq = __unwind_incomplete_requests(engine); + if (rq && i915_request_started(rq)) + __i915_request_reset(rq, stalled); + + cancel_port_requests(engine); + + /* Clear the global submission state, we will submit from scratch */ + intel_ring_reset(engine->legacy.ring, 0); + set_current_context(&engine->legacy.context, NULL); +} + +static void ring_reset_rewind(struct intel_engine_cs *engine, bool stalled) +{ + unsigned long flags; + + spin_lock_irqsave(&engine->active.lock, flags); + __ring_rewind(engine, stalled); + spin_unlock_irqrestore(&engine->active.lock, flags); +} + +static void nop_submission_tasklet(unsigned long data) +{ + struct intel_engine_cs * const engine = (struct intel_engine_cs *)data; + + /* The driver is wedged; don't process any more events. */ + WRITE_ONCE(engine->execlists.queue_priority_hint, INT_MIN); +} + +static void ring_reset_cancel(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request *rq, *rn; + unsigned long flags; + struct rb_node *rb; + + spin_lock_irqsave(&engine->active.lock, flags); + + __ring_rewind(engine, true); + + /* Mark all submitted requests as skipped. */ + list_for_each_entry(rq, &engine->active.requests, sched.link) { + i915_request_set_error_once(rq, -EIO); + i915_request_mark_complete(rq); + } + + /* Flush the queued requests to the timeline list (for retiring). */ + while ((rb = rb_first_cached(&el->queue))) { + struct i915_priolist *p = to_priolist(rb); + int i; + + priolist_for_each_request_consume(rq, rn, p, i) { + i915_request_set_error_once(rq, -EIO); + i915_request_mark_complete(rq); + __i915_request_submit(rq); + } + + rb_erase_cached(&p->node, &el->queue); + i915_priolist_free(p); + } + + el->queue_priority_hint = INT_MIN; + el->queue = RB_ROOT_CACHED; + + /* Remaining _unready_ requests will be nop'ed when submitted */ + + GEM_BUG_ON(__tasklet_is_enabled(&el->tasklet)); + el->tasklet.func = nop_submission_tasklet; + + spin_unlock_irqrestore(&engine->active.lock, flags); +} + +static void reset_finish(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + + intel_ring_submission_reset_finish(engine); + + if (__tasklet_enable(&el->tasklet)) + tasklet_hi_schedule(&el->tasklet); +} + +static u32 *ring_map(struct intel_ring *ring, u32 len) +{ + u32 *va; + + if (unlikely(ring->tail + len > ring->effective_size)) { + memset(ring->vaddr + ring->tail, 0, ring->size - ring->tail); + ring->tail = 0; + } + + va = ring->vaddr + ring->tail; + ring->tail = intel_ring_wrap(ring, ring->tail + len); + + return va; +} + +static inline u32 *ring_map_dw(struct intel_ring *ring, u32 len) +{ + return ring_map(ring, len * sizeof(u32)); +} + +static void ring_copy(struct intel_ring *dst, + const struct intel_ring *src, + u32 start, u32 end) +{ + unsigned int len; + void *out; + + len = end - start; + if (end < start) + len += src->size; + out = ring_map(dst, len); + + if (end < start) { + len = src->size - start; + memcpy(out, src->vaddr + start, len); + out += len; + start = 0; + } + + memcpy(out, src->vaddr + start, end - start); +} + +static void switch_context(struct intel_ring *ring, struct i915_request *rq) +{ +} + +static struct i915_request *ring_submit(struct i915_request *rq) +{ + struct intel_ring *ring = rq->engine->legacy.ring; + + __i915_request_submit(rq); + + if (rq->engine->legacy.context != rq->context) { + switch_context(ring, rq); + set_current_context(&rq->engine->legacy.context, rq->context); + } + + ring_copy(ring, rq->ring, rq->head, rq->tail); + return rq; +} + +static struct i915_request ** +copy_active(struct i915_request **port, struct i915_request * const *active) +{ + while (*active) + *port++ = *active++; + + return port; +} + +static void __dequeue(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request ** const last_port = el->pending + el->port_mask; + struct i915_request **port, *last; + struct rb_node *rb; + + lockdep_assert_held(&engine->active.lock); + + port = copy_active(el->pending, el->active); + if (port > last_port) + return; + + last = NULL; + while ((rb = rb_first_cached(&el->queue))) { + struct i915_priolist *p = to_priolist(rb); + struct i915_request *rq, *rn; + int i; + + priolist_for_each_request_consume(rq, rn, p, i) { + GEM_BUG_ON(rq == last); + if (last && rq->context != last->context) { + if (port == last_port) + goto done; + + *port++ = schedule_in(engine, last); + } + + last = ring_submit(rq); + } + + rb_erase_cached(&p->node, &el->queue); + i915_priolist_free(p); + } + +done: + el->queue_priority_hint = queue_prio(rb); + if (last) { + *port++ = schedule_in(engine, last); + *port++ = NULL; + WRITE_ONCE(el->active, el->pending); + + wmb(); /* paranoid flush of WCB before RING_TAIL write */ + ENGINE_WRITE(engine, RING_TAIL, engine->legacy.ring->tail); + memcpy(el->inflight, el->pending, + (port - el->pending) * sizeof(*port)); + + WRITE_ONCE(el->active, el->inflight); + GEM_BUG_ON(!*el->active); + } +} + +static void __submission_tasklet(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + struct i915_request *rq; + + while ((rq = *el->active)) { + if (!i915_request_completed(rq)) + break; + + schedule_out(engine, rq); + el->active++; + } + + if (el->queue_priority_hint != INT_MIN) + __dequeue(engine); +} + +static void submission_tasklet(unsigned long data) +{ + struct intel_engine_cs * const engine = (struct intel_engine_cs *)data; + unsigned long flags; + + spin_lock_irqsave(&engine->active.lock, flags); + __submission_tasklet(engine); + spin_unlock_irqrestore(&engine->active.lock, flags); +} + +static void queue_request(struct intel_engine_cs *engine, + struct i915_request *rq) +{ + GEM_BUG_ON(!list_empty(&rq->sched.link)); + list_add_tail(&rq->sched.link, + i915_sched_lookup_priolist(engine, rq_prio(rq))); + set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags); +} + +static void __submit_queue_imm(struct intel_engine_cs *engine) +{ + struct intel_engine_execlists * const el = &engine->execlists; + + if (reset_in_progress(el)) + return; /* defer until we restart the engine following reset */ + + __submission_tasklet(engine); +} + +static void submit_queue(struct intel_engine_cs *engine, + const struct i915_request *rq) +{ + struct intel_engine_execlists *el = &engine->execlists; + + if (rq_prio(rq) <= el->queue_priority_hint) + return; + + el->queue_priority_hint = rq_prio(rq); + __submit_queue_imm(engine); +} + +static void submit_request(struct i915_request *rq) +{ + struct intel_engine_cs *engine = rq->engine; + unsigned long flags; + + spin_lock_irqsave(&engine->active.lock, flags); + + queue_request(engine, rq); + + GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root)); + GEM_BUG_ON(list_empty(&rq->sched.link)); + + submit_queue(engine, rq); + + spin_unlock_irqrestore(&engine->active.lock, flags); +} + +static void submission_park(struct intel_engine_cs *engine) +{ + GEM_BUG_ON(engine->execlists.queue_priority_hint != INT_MIN); + intel_engine_unpin_breadcrumbs_irq(engine); + submission_tasklet((unsigned long)engine); /* drain the submit queue */ +} + +static void submission_unpark(struct intel_engine_cs *engine) +{ + intel_engine_pin_breadcrumbs_irq(engine); +} + +static void ring_context_destroy(struct kref *ref) +{ + struct intel_context *ce = container_of(ref, typeof(*ce), ref); + + GEM_BUG_ON(intel_context_is_pinned(ce)); + + if (ce->state) + i915_vma_put(ce->state); + if (test_bit(CONTEXT_ALLOC_BIT, &ce->flags)) + intel_ring_put(ce->ring); + + intel_context_fini(ce); + intel_context_free(ce); +} + +static void ring_context_unpin(struct intel_context *ce) +{ +} + +static int alloc_context_vma(struct intel_context *ce) + +{ + struct intel_engine_cs *engine = ce->engine; + struct drm_i915_gem_object *obj; + struct i915_vma *vma; + int err; + + obj = i915_gem_object_create_shmem(engine->i915, engine->context_size); + if (IS_ERR(obj)) + return PTR_ERR(obj); + + /* + * Try to make the context utilize L3 as well as LLC. + * + * On VLV we don't have L3 controls in the PTEs so we + * shouldn't touch the cache level, especially as that + * would make the object snooped which might have a + * negative performance impact. + * + * Snooping is required on non-llc platforms in execlist + * mode, but since all GGTT accesses use PAT entry 0 we + * get snooping anyway regardless of cache_level. + * + * This is only applicable for Ivy Bridge devices since + * later platforms don't have L3 control bits in the PTE. + */ + if (IS_IVYBRIDGE(engine->i915)) + i915_gem_object_set_cache_coherency(obj, I915_CACHE_L3_LLC); + + if (engine->default_state) { + void *vaddr; + + vaddr = i915_gem_object_pin_map(obj, I915_MAP_WB); + if (IS_ERR(vaddr)) { + err = PTR_ERR(vaddr); + goto err_obj; + } + + shmem_read(engine->default_state, 0, + vaddr, engine->context_size); + __set_bit(CONTEXT_VALID_BIT, &ce->flags); + + i915_gem_object_flush_map(obj); + i915_gem_object_unpin_map(obj); + } + + vma = i915_vma_instance(obj, &engine->gt->ggtt->vm, NULL); + if (IS_ERR(vma)) { + err = PTR_ERR(vma); + goto err_obj; + } + + ce->state = vma; + return 0; + +err_obj: + i915_gem_object_put(obj); + return err; +} + +static int alloc_timeline(struct intel_context *ce) +{ + struct intel_engine_cs *engine = ce->engine; + struct intel_timeline *tl; + struct i915_vma *hwsp; + + /* + * Use the static global HWSP for the kernel context, and + * a dynamically allocated cacheline for everyone else. + */ + hwsp = NULL; + if (unlikely(intel_context_is_barrier(ce))) + hwsp = engine->status_page.vma; + + tl = intel_timeline_create(engine->gt, hwsp); + if (IS_ERR(tl)) + return PTR_ERR(tl); + + ce->timeline = tl; + return 0; +} + +static int ring_context_alloc(struct intel_context *ce) +{ + struct intel_engine_cs *engine = ce->engine; + struct intel_ring *ring; + int err; + + GEM_BUG_ON(ce->state); + if (engine->context_size) { + err = alloc_context_vma(ce); + if (err) + return err; + } + + if (!ce->timeline) { + err = alloc_timeline(ce); + if (err) + goto err_vma; + } + + ring = intel_engine_create_ring(engine, + (unsigned long)ce->ring | + INTEL_RING_CREATE_INTERNAL); + if (IS_ERR(ring)) { + err = PTR_ERR(ring); + goto err_timeline; + } + ce->ring = ring; + + return 0; + +err_timeline: + intel_timeline_put(ce->timeline); +err_vma: + if (ce->state) { + i915_vma_put(ce->state); + ce->state = NULL; + } + return err; +} + +static int ring_context_pin(struct intel_context *ce) +{ + return 0; +} + +static void ring_context_reset(struct intel_context *ce) +{ + intel_ring_reset(ce->ring, 0); + clear_bit(CONTEXT_VALID_BIT, &ce->flags); +} + +static const struct intel_context_ops ring_context_ops = { + .alloc = ring_context_alloc, + + .pin = ring_context_pin, + .unpin = ring_context_unpin, + + .enter = intel_context_enter_engine, + .exit = intel_context_exit_engine, + + .reset = ring_context_reset, + .destroy = ring_context_destroy, +}; + +static int ring_request_alloc(struct i915_request *rq) +{ + int ret; + + GEM_BUG_ON(!intel_context_is_pinned(rq->context)); + + /* + * Flush enough space to reduce the likelihood of waiting after + * we start building the request - in which case we will just + * have to repeat work. + */ + rq->reserved_space += LEGACY_REQUEST_SIZE; + + /* Unconditionally invalidate GPU caches and TLBs. */ + ret = rq->engine->emit_flush(rq, EMIT_INVALIDATE); + if (ret) + return ret; + + rq->reserved_space -= LEGACY_REQUEST_SIZE; + return 0; +} + +static void set_default_submission(struct intel_engine_cs *engine) +{ + engine->submit_request = submit_request; + engine->execlists.tasklet.func = submission_tasklet; +} + +static void ring_release(struct intel_engine_cs *engine) +{ + intel_engine_cleanup_common(engine); + + set_current_context(&engine->legacy.context, NULL); + + intel_ring_unpin(engine->legacy.ring); + intel_ring_put(engine->legacy.ring); +} + +static void setup_irq(struct intel_engine_cs *engine) +{ +} + +static void setup_common(struct intel_engine_cs *engine) +{ + struct drm_i915_private *i915 = engine->i915; + + /* gen8+ are only supported with execlists */ + GEM_BUG_ON(INTEL_GEN(i915) >= 8); + GEM_BUG_ON(INTEL_GEN(i915) < 8); + + setup_irq(engine); + + engine->park = submission_park; + engine->unpark = submission_unpark; + engine->schedule = i915_schedule; + + engine->resume = intel_ring_submission_resume; + engine->reset.prepare = reset_prepare; + engine->reset.rewind = ring_reset_rewind; + engine->reset.cancel = ring_reset_cancel; + engine->reset.finish = reset_finish; + + engine->cops = &ring_context_ops; + engine->request_alloc = ring_request_alloc; + + engine->set_default_submission = set_default_submission; +} + +static void setup_rcs(struct intel_engine_cs *engine) +{ +} + +static void setup_vcs(struct intel_engine_cs *engine) +{ +} + +static void setup_bcs(struct intel_engine_cs *engine) +{ +} + +static void setup_vecs(struct intel_engine_cs *engine) +{ + GEM_BUG_ON(!IS_HASWELL(engine->i915)); +} + +static unsigned int global_ring_size(void) +{ + /* Enough space to hold 2 clients and the context switch */ + return roundup_pow_of_two(EXECLIST_MAX_PORTS * SZ_16K + SZ_4K); +} + +int intel_ring_scheduler_setup(struct intel_engine_cs *engine) +{ + struct intel_ring *ring; + int err; + + GEM_BUG_ON(HAS_EXECLISTS(engine->i915)); + + tasklet_init(&engine->execlists.tasklet, + submission_tasklet, (unsigned long)engine); + + setup_common(engine); + + switch (engine->class) { + case RENDER_CLASS: + setup_rcs(engine); + break; + case VIDEO_DECODE_CLASS: + setup_vcs(engine); + break; + case COPY_ENGINE_CLASS: + setup_bcs(engine); + break; + case VIDEO_ENHANCEMENT_CLASS: + setup_vecs(engine); + break; + default: + MISSING_CASE(engine->class); + return -ENODEV; + } + + ring = intel_engine_create_ring(engine, global_ring_size()); + if (IS_ERR(ring)) { + err = PTR_ERR(ring); + goto err; + } + + err = intel_ring_pin(ring); + if (err) + goto err_ring; + + GEM_BUG_ON(engine->legacy.ring); + engine->legacy.ring = ring; + + engine->flags |= I915_ENGINE_NEEDS_BREADCRUMB_TASKLET; + + /* Finally, take ownership and responsibility for cleanup! */ + engine->release = ring_release; + return 0; + +err_ring: + intel_ring_put(ring); +err: + intel_engine_cleanup_common(engine); + return err; +} diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c index 68a08486fc87..4cf7c6486223 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c @@ -36,6 +36,7 @@ #include "intel_gt.h" #include "intel_reset.h" #include "intel_ring.h" +#include "intel_ring_submission.h" #include "shmem_utils.h" /* Rough estimate of the typical request size, performing a flush, @@ -214,7 +215,7 @@ static void set_pp_dir(struct intel_engine_cs *engine) } } -static int xcs_resume(struct intel_engine_cs *engine) +int intel_ring_submission_resume(struct intel_engine_cs *engine) { struct drm_i915_private *dev_priv = engine->i915; struct intel_ring *ring = engine->legacy.ring; @@ -318,7 +319,7 @@ static int xcs_resume(struct intel_engine_cs *engine) return ret; } -static void reset_prepare(struct intel_engine_cs *engine) +void intel_ring_submission_reset_prepare(struct intel_engine_cs *engine) { struct intel_uncore *uncore = engine->uncore; const u32 base = engine->mmio_base; @@ -425,7 +426,7 @@ static void reset_rewind(struct intel_engine_cs *engine, bool stalled) spin_unlock_irqrestore(&engine->active.lock, flags); } -static void reset_finish(struct intel_engine_cs *engine) +void intel_ring_submission_reset_finish(struct intel_engine_cs *engine) { } @@ -1056,11 +1057,11 @@ static void setup_common(struct intel_engine_cs *engine) setup_irq(engine); - engine->resume = xcs_resume; - engine->reset.prepare = reset_prepare; + engine->resume = intel_ring_submission_resume; + engine->reset.prepare = intel_ring_submission_reset_prepare; engine->reset.rewind = reset_rewind; engine->reset.cancel = reset_cancel; - engine->reset.finish = reset_finish; + engine->reset.finish = intel_ring_submission_reset_finish; engine->cops = &ring_context_ops; engine->request_alloc = ring_request_alloc; diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.h b/drivers/gpu/drm/i915/gt/intel_ring_submission.h new file mode 100644 index 000000000000..701eb033e055 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_RING_SUBMISSION_H__ +#define __INTEL_RING_SUBMISSION_H__ + +struct intel_engine_cs; + +void intel_ring_submission_reset_prepare(struct intel_engine_cs *engine); +void intel_ring_submission_reset_finish(struct intel_engine_cs *engine); + +int intel_ring_submission_resume(struct intel_engine_cs *engine); + +#endif /* __INTEL_RING_SUBMISSION_H__ */ From patchwork Fri Jun 5 12:23:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589573 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC89D90 for ; Fri, 5 Jun 2020 12:23:58 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A55C4206E6 for ; Fri, 5 Jun 2020 12:23:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A55C4206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A58D76E8AA; Fri, 5 Jun 2020 12:23:52 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 018526E119 for ; Fri, 5 Jun 2020 12:23:47 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406196-1500050 for multiple; Fri, 05 Jun 2020 13:23:39 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:32 +0100 Message-Id: <20200605122334.2798-8-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 08/10] drm/i915/gt: Enable busy-stats for ring-scheduler X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Couple up the context in/out accounting to record how long each engine is busy handling requests. This is exposed to userspace for more accurate measurements, and also enables our soft-rps timer. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/gt/intel_engine_stats.h | 49 ++++++++++ drivers/gpu/drm/i915/gt/intel_lrc.c | 34 +------ .../gpu/drm/i915/gt/intel_ring_scheduler.c | 4 + drivers/gpu/drm/i915/gt/selftest_engine_pm.c | 90 +++++++++++++++++++ drivers/gpu/drm/i915/gt/selftest_rps.c | 5 ++ 5 files changed, 149 insertions(+), 33 deletions(-) create mode 100644 drivers/gpu/drm/i915/gt/intel_engine_stats.h diff --git a/drivers/gpu/drm/i915/gt/intel_engine_stats.h b/drivers/gpu/drm/i915/gt/intel_engine_stats.h new file mode 100644 index 000000000000..58491eae3482 --- /dev/null +++ b/drivers/gpu/drm/i915/gt/intel_engine_stats.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright © 2020 Intel Corporation + */ + +#ifndef __INTEL_ENGINE_STATS_H__ +#define __INTEL_ENGINE_STATS_H__ + +#include +#include +#include + +#include "i915_gem.h" /* GEM_BUG_ON */ +#include "intel_engine.h" + +static inline void intel_engine_context_in(struct intel_engine_cs *engine) +{ + unsigned long flags; + + if (atomic_add_unless(&engine->stats.active, 1, 0)) + return; + + write_seqlock_irqsave(&engine->stats.lock, flags); + if (!atomic_add_unless(&engine->stats.active, 1, 0)) { + engine->stats.start = ktime_get(); + atomic_inc(&engine->stats.active); + } + write_sequnlock_irqrestore(&engine->stats.lock, flags); +} + +static inline void intel_engine_context_out(struct intel_engine_cs *engine) +{ + unsigned long flags; + + GEM_BUG_ON(!atomic_read(&engine->stats.active)); + + if (atomic_add_unless(&engine->stats.active, -1, 1)) + return; + + write_seqlock_irqsave(&engine->stats.lock, flags); + if (atomic_dec_and_test(&engine->stats.active)) { + engine->stats.total = + ktime_add(engine->stats.total, + ktime_sub(ktime_get(), engine->stats.start)); + } + write_sequnlock_irqrestore(&engine->stats.lock, flags); +} + +#endif /* __INTEL_ENGINE_STATS_H__ */ diff --git a/drivers/gpu/drm/i915/gt/intel_lrc.c b/drivers/gpu/drm/i915/gt/intel_lrc.c index d55a5e0466e5..36e2ce4ac2fa 100644 --- a/drivers/gpu/drm/i915/gt/intel_lrc.c +++ b/drivers/gpu/drm/i915/gt/intel_lrc.c @@ -139,6 +139,7 @@ #include "i915_vgpu.h" #include "intel_context.h" #include "intel_engine_pm.h" +#include "intel_engine_stats.h" #include "intel_gt.h" #include "intel_gt_pm.h" #include "intel_gt_requests.h" @@ -1187,39 +1188,6 @@ execlists_context_status_change(struct i915_request *rq, unsigned long status) status, rq); } -static void intel_engine_context_in(struct intel_engine_cs *engine) -{ - unsigned long flags; - - if (atomic_add_unless(&engine->stats.active, 1, 0)) - return; - - write_seqlock_irqsave(&engine->stats.lock, flags); - if (!atomic_add_unless(&engine->stats.active, 1, 0)) { - engine->stats.start = ktime_get(); - atomic_inc(&engine->stats.active); - } - write_sequnlock_irqrestore(&engine->stats.lock, flags); -} - -static void intel_engine_context_out(struct intel_engine_cs *engine) -{ - unsigned long flags; - - GEM_BUG_ON(!atomic_read(&engine->stats.active)); - - if (atomic_add_unless(&engine->stats.active, -1, 1)) - return; - - write_seqlock_irqsave(&engine->stats.lock, flags); - if (atomic_dec_and_test(&engine->stats.active)) { - engine->stats.total = - ktime_add(engine->stats.total, - ktime_sub(ktime_get(), engine->stats.start)); - } - write_sequnlock_irqrestore(&engine->stats.lock, flags); -} - static void execlists_check_context(const struct intel_context *ce, const struct intel_engine_cs *engine) diff --git a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c index c8cd435d1c51..aaff554865b1 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c +++ b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c @@ -9,6 +9,7 @@ #include "i915_drv.h" #include "intel_context.h" +#include "intel_engine_stats.h" #include "intel_gt.h" #include "intel_gt_pm.h" #include "intel_gt_requests.h" @@ -59,6 +60,7 @@ static struct i915_request * schedule_in(struct intel_engine_cs *engine, struct i915_request *rq) { __intel_gt_pm_get(engine->gt); + intel_engine_context_in(engine); return i915_request_get(rq); } @@ -71,6 +73,7 @@ schedule_out(struct intel_engine_cs *engine, struct i915_request *rq) intel_engine_add_retire(engine, ce->timeline); i915_request_put(rq); + intel_engine_context_out(engine); intel_gt_pm_put_async(engine->gt); } @@ -747,6 +750,7 @@ int intel_ring_scheduler_setup(struct intel_engine_cs *engine) engine->legacy.ring = ring; engine->flags |= I915_ENGINE_NEEDS_BREADCRUMB_TASKLET; + engine->flags |= I915_ENGINE_SUPPORTS_STATS; /* Finally, take ownership and responsibility for cleanup! */ engine->release = ring_release; diff --git a/drivers/gpu/drm/i915/gt/selftest_engine_pm.c b/drivers/gpu/drm/i915/gt/selftest_engine_pm.c index cbf6b0735272..fef9709b7cfa 100644 --- a/drivers/gpu/drm/i915/gt/selftest_engine_pm.c +++ b/drivers/gpu/drm/i915/gt/selftest_engine_pm.c @@ -7,6 +7,95 @@ #include "i915_selftest.h" #include "selftest_engine.h" #include "selftests/igt_atomic.h" +#include "selftests/igt_flush_test.h" +#include "selftests/igt_spinner.h" + +static int live_engine_busy_stats(void *arg) +{ + struct intel_gt *gt = arg; + struct intel_engine_cs *engine; + enum intel_engine_id id; + struct igt_spinner spin; + int err = 0; + + /* + * Check that if an engine supports busy-stats, they tell the truth. + */ + + if (igt_spinner_init(&spin, gt)) + return -ENOMEM; + + GEM_BUG_ON(intel_gt_pm_is_awake(gt)); + for_each_engine(engine, gt, id) { + struct i915_request *rq; + ktime_t dt, de; + + if (!intel_engine_supports_stats(engine)) + continue; + + if (!intel_engine_can_store_dword(engine)) + continue; + + if (intel_gt_pm_wait_for_idle(gt)) { + err = -EBUSY; + break; + } + + preempt_disable(); + dt = ktime_get(); + de = intel_engine_get_busy_time(engine); + udelay(100); + dt = ktime_sub(ktime_get(), dt); + de = ktime_sub(intel_engine_get_busy_time(engine), de); + preempt_enable(); + if (de > 10) { + pr_err("%s: reported %lldns [%d%%] busyness while sleeping [for %lldns]\n", + engine->name, + de, (int)div64_u64(100 * de, dt), dt); + err = -EINVAL; + break; + } + + rq = igt_spinner_create_request(&spin, + engine->kernel_context, + MI_NOOP); + if (IS_ERR(rq)) { + err = PTR_ERR(rq); + break; + } + i915_request_add(rq); + + if (!igt_wait_for_spinner(&spin, rq)) { + intel_gt_set_wedged(engine->gt); + err = -ETIME; + break; + } + + preempt_disable(); + dt = ktime_get(); + de = intel_engine_get_busy_time(engine); + udelay(100); + dt = ktime_sub(ktime_get(), dt); + de = ktime_sub(intel_engine_get_busy_time(engine), de); + preempt_enable(); + if (100 * de < 95 * dt) { + pr_err("%s: reported only %lldns [%d%%] busyness while spinning [for %lldns]\n", + engine->name, + de, (int)div64_u64(100 * de, dt), dt); + err = -EINVAL; + break; + } + + igt_spinner_end(&spin); + if (igt_flush_test(gt->i915)) { + err = -EIO; + break; + } + } + + igt_spinner_fini(&spin); + return err; +} static int live_engine_pm(void *arg) { @@ -77,6 +166,7 @@ static int live_engine_pm(void *arg) int live_engine_pm_selftests(struct intel_gt *gt) { static const struct i915_subtest tests[] = { + SUBTEST(live_engine_busy_stats), SUBTEST(live_engine_pm), }; diff --git a/drivers/gpu/drm/i915/gt/selftest_rps.c b/drivers/gpu/drm/i915/gt/selftest_rps.c index 5049c3dd08a6..5e364fb31aea 100644 --- a/drivers/gpu/drm/i915/gt/selftest_rps.c +++ b/drivers/gpu/drm/i915/gt/selftest_rps.c @@ -1252,6 +1252,11 @@ int live_rps_dynamic(void *arg) if (igt_spinner_init(&spin, gt)) return -ENOMEM; + if (intel_rps_has_interrupts(rps)) + pr_info("RPS has interrupt support\n"); + if (intel_rps_uses_timer(rps)) + pr_info("RPS has timer support\n"); + for_each_engine(engine, gt, id) { struct i915_request *rq; struct { From patchwork Fri Jun 5 12:23:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589559 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B59790 for ; Fri, 5 Jun 2020 12:23:50 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E71CD206E6 for ; Fri, 5 Jun 2020 12:23:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E71CD206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7E5496E0A5; Fri, 5 Jun 2020 12:23:48 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id 17E666E0A5 for ; Fri, 5 Jun 2020 12:23:46 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406197-1500050 for multiple; Fri, 05 Jun 2020 13:23:39 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:33 +0100 Message-Id: <20200605122334.2798-9-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 09/10] drm/i915/gt: Implement ring scheduler for gen6/7 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" A key prolem with legacy ring buffer submission is that it is an inheret FIFO queue across all clients; if one blocks, they all block. A scheduler allows us to avoid that limitation, and ensures that all clients can submit in parallel, removing the resource contention of the global ringbuffer. Having built the ring scheduler infrastructure over top of the global ringbuffer submission, we now need to provide the HW knowledge required to build command packets and implement context switching. Signed-off-by: Chris Wilson --- .../gpu/drm/i915/gt/intel_ring_scheduler.c | 412 +++++++++++++++++- 1 file changed, 410 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c index aaff554865b1..463c836cbde2 100644 --- a/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c +++ b/drivers/gpu/drm/i915/gt/intel_ring_scheduler.c @@ -7,6 +7,10 @@ #include +#include "gen2_engine_cs.h" +#include "gen6_engine_cs.h" +#include "gen6_ppgtt.h" +#include "gen7_renderclear.h" #include "i915_drv.h" #include "intel_context.h" #include "intel_engine_stats.h" @@ -282,8 +286,258 @@ static void ring_copy(struct intel_ring *dst, memcpy(out, src->vaddr + start, end - start); } +static void mi_set_context(struct intel_ring *ring, + struct intel_engine_cs *engine, + struct intel_context *ce, + u32 flags) +{ + struct drm_i915_private *i915 = engine->i915; + enum intel_engine_id id; + const int num_engines = + IS_HASWELL(i915) ? RUNTIME_INFO(i915)->num_engines - 1 : 0; + int len; + u32 *cs; + + len = 4; + if (IS_GEN(i915, 7)) + len += 2 + (num_engines ? 4 * num_engines + 6 : 0); + else if (IS_GEN(i915, 5)) + len += 2; + + cs = ring_map_dw(ring, len); + + /* WaProgramMiArbOnOffAroundMiSetContext:ivb,vlv,hsw,bdw,chv */ + if (IS_GEN(i915, 7)) { + *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE; + if (num_engines) { + struct intel_engine_cs *signaller; + + *cs++ = MI_LOAD_REGISTER_IMM(num_engines); + for_each_engine(signaller, engine->gt, id) { + if (signaller == engine) + continue; + + *cs++ = i915_mmio_reg_offset( + RING_PSMI_CTL(signaller->mmio_base)); + *cs++ = _MASKED_BIT_ENABLE( + GEN6_PSMI_SLEEP_MSG_DISABLE); + } + } + } else if (IS_GEN(i915, 5)) { + /* + * This w/a is only listed for pre-production ilk a/b steppings, + * but is also mentioned for programming the powerctx. To be + * safe, just apply the workaround; we do not use SyncFlush so + * this should never take effect and so be a no-op! + */ + *cs++ = MI_SUSPEND_FLUSH | MI_SUSPEND_FLUSH_EN; + } + + *cs++ = MI_NOOP; + *cs++ = MI_SET_CONTEXT; + *cs++ = i915_ggtt_offset(ce->state) | flags; + /* + * w/a: MI_SET_CONTEXT must always be followed by MI_NOOP + * WaMiSetContext_Hang:snb,ivb,vlv + */ + *cs++ = MI_NOOP; + + if (IS_GEN(i915, 7)) { + if (num_engines) { + struct intel_engine_cs *signaller; + i915_reg_t last_reg = {}; /* keep gcc quiet */ + + *cs++ = MI_LOAD_REGISTER_IMM(num_engines); + for_each_engine(signaller, engine->gt, id) { + if (signaller == engine) + continue; + + last_reg = RING_PSMI_CTL(signaller->mmio_base); + *cs++ = i915_mmio_reg_offset(last_reg); + *cs++ = _MASKED_BIT_DISABLE( + GEN6_PSMI_SLEEP_MSG_DISABLE); + } + + /* Insert a delay before the next switch! */ + *cs++ = MI_STORE_REGISTER_MEM | MI_SRM_LRM_GLOBAL_GTT; + *cs++ = i915_mmio_reg_offset(last_reg); + *cs++ = intel_gt_scratch_offset(engine->gt, + INTEL_GT_SCRATCH_FIELD_DEFAULT); + *cs++ = MI_NOOP; + } + *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE; + } else if (IS_GEN(i915, 5)) { + *cs++ = MI_SUSPEND_FLUSH; + } +} + +static struct i915_address_space *vm_alias(struct i915_address_space *vm) +{ + if (i915_is_ggtt(vm)) + vm = &i915_vm_to_ggtt(vm)->alias->vm; + + return vm; +} + +static void load_pd_dir(struct intel_ring *ring, + struct intel_engine_cs *engine, + const struct i915_ppgtt *ppgtt) +{ + u32 *cs = ring_map_dw(ring, 10); + + *cs++ = MI_LOAD_REGISTER_IMM(1); + *cs++ = i915_mmio_reg_offset(RING_PP_DIR_DCLV(engine->mmio_base)); + *cs++ = PP_DIR_DCLV_2G; + + *cs++ = MI_LOAD_REGISTER_IMM(1); + *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine->mmio_base)); + *cs++ = px_base(ppgtt->pd)->ggtt_offset << 10; + + /* Stall until the page table load is complete? */ + *cs++ = MI_STORE_REGISTER_MEM | MI_SRM_LRM_GLOBAL_GTT; + *cs++ = i915_mmio_reg_offset(RING_PP_DIR_BASE(engine->mmio_base)); + *cs++ = intel_gt_scratch_offset(engine->gt, + INTEL_GT_SCRATCH_FIELD_DEFAULT); + *cs++ = MI_NOOP; +} + +static struct i915_address_space *current_vm(struct intel_engine_cs *engine) +{ + struct intel_context *old = engine->legacy.context; + + return old ? vm_alias(old->vm) : NULL; +} + +static void gen6_emit_invalidate_rcs(struct intel_ring *ring, + struct intel_engine_cs *engine) +{ + u32 addr, flags; + u32 *cs; + + addr = intel_gt_scratch_offset(engine->gt, + INTEL_GT_SCRATCH_FIELD_RENDER_FLUSH); + + flags = PIPE_CONTROL_QW_WRITE | PIPE_CONTROL_CS_STALL; + flags |= PIPE_CONTROL_TLB_INVALIDATE; + + if (INTEL_GEN(engine->i915) >= 7) + flags |= PIPE_CONTROL_GLOBAL_GTT_IVB; + else + addr |= PIPE_CONTROL_GLOBAL_GTT; + + cs = ring_map_dw(ring, 4); + *cs++ = GFX_OP_PIPE_CONTROL(4); + *cs++ = flags; + *cs++ = addr; + *cs++ = 0; +} + +static struct i915_address_space * +clear_residuals(struct intel_ring *ring, struct intel_engine_cs *engine) +{ + struct intel_context *ce = engine->kernel_context; + struct i915_address_space *vm = vm_alias(engine->gt->vm); + u32 flags; + + if (vm != current_vm(engine)) + load_pd_dir(ring, engine, i915_vm_to_ppgtt(vm)); + + if (ce->state) + mi_set_context(ring, engine, ce, + MI_MM_SPACE_GTT | MI_RESTORE_INHIBIT); + + if (IS_HASWELL(engine->i915)) + flags = MI_BATCH_PPGTT_HSW | MI_BATCH_NON_SECURE_HSW; + else + flags = MI_BATCH_NON_SECURE_I965; + + __gen6_emit_bb_start(ring_map_dw(ring, 2), + engine->wa_ctx.vma->node.start, flags); + + return vm; +} + +static void remap_l3_slice(struct intel_ring *ring, + struct intel_engine_cs *engine, + int slice) +{ + u32 *cs, *remap_info = engine->i915->l3_parity.remap_info[slice]; + int i; + + if (!remap_info) + return; + + /* + * Note: We do not worry about the concurrent register cacheline hang + * here because no other code should access these registers other than + * at initialization time. + */ + cs = ring_map_dw(ring, GEN7_L3LOG_SIZE / 4 * 2 + 2); + *cs++ = MI_LOAD_REGISTER_IMM(GEN7_L3LOG_SIZE / 4); + for (i = 0; i < GEN7_L3LOG_SIZE / 4; i++) { + *cs++ = i915_mmio_reg_offset(GEN7_L3LOG(slice, i)); + *cs++ = remap_info[i]; + } + *cs++ = MI_NOOP; +} + +static void remap_l3(struct intel_ring *ring, + struct intel_engine_cs *engine, + struct intel_context *ce) +{ + struct i915_gem_context *ctx = + rcu_dereference_protected(ce->gem_context, true); + int bit, idx = -1; + + if (!ctx || !ctx->remap_slice) + return; + + do { + bit = ffs(ctx->remap_slice); + remap_l3_slice(ring, engine, idx += bit); + } while (ctx->remap_slice >>= bit); +} + static void switch_context(struct intel_ring *ring, struct i915_request *rq) { + struct intel_engine_cs *engine = rq->engine; + struct i915_address_space *cvm = current_vm(engine); + struct intel_context *ce = rq->context; + struct i915_address_space *vm; + + if (engine->wa_ctx.vma && ce != engine->kernel_context) { + if (engine->wa_ctx.vma->private != ce) { + cvm = clear_residuals(ring, engine); + intel_context_put(engine->wa_ctx.vma->private); + engine->wa_ctx.vma->private = intel_context_get(ce); + } + } + + vm = vm_alias(ce->vm); + if (vm != cvm) + load_pd_dir(ring, engine, i915_vm_to_ppgtt(vm)); + + if (ce->state) { + u32 flags; + + GEM_BUG_ON(engine->id != RCS0); + + /* For resource streamer on HSW+ and power context elsewhere */ + BUILD_BUG_ON(HSW_MI_RS_SAVE_STATE_EN != MI_SAVE_EXT_STATE_EN); + BUILD_BUG_ON(HSW_MI_RS_RESTORE_STATE_EN != MI_RESTORE_EXT_STATE_EN); + + flags = MI_SAVE_EXT_STATE_EN | MI_MM_SPACE_GTT; + if (test_bit(CONTEXT_VALID_BIT, &ce->flags)) { + gen6_emit_invalidate_rcs(ring, engine); + flags |= MI_RESTORE_EXT_STATE_EN; + } else { + flags |= MI_RESTORE_INHIBIT; + } + + mi_set_context(ring, engine, ce, flags); + } + + remap_l3(ring, engine, ce); } static struct i915_request *ring_submit(struct i915_request *rq) @@ -449,6 +703,33 @@ static void submission_unpark(struct intel_engine_cs *engine) intel_engine_pin_breadcrumbs_irq(engine); } +static int gen6_emit_init_breadcrumb(struct i915_request *rq) +{ + struct intel_timeline *tl = i915_request_timeline(rq); + u32 *cs; + + GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq)); + if (!tl->has_initial_breadcrumb) + return 0; + + cs = intel_ring_begin(rq, 4); + if (IS_ERR(cs)) + return PTR_ERR(cs); + + *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT; + *cs++ = 0; + *cs++ = tl->hwsp_offset; + *cs++ = rq->fence.seqno - 1; + + intel_ring_advance(rq, cs); + + /* Record the updated position of the request's payload */ + rq->infix = intel_ring_offset(rq, cs); + + __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags); + return 0; +} + static void ring_context_destroy(struct kref *ref) { struct intel_context *ce = container_of(ref, typeof(*ce), ref); @@ -464,8 +745,30 @@ static void ring_context_destroy(struct kref *ref) intel_context_free(ce); } +static int __context_pin_ppgtt(struct intel_context *ce) +{ + struct i915_address_space *vm; + int err = 0; + + vm = vm_alias(ce->vm); + if (vm) + err = gen6_ppgtt_pin(i915_vm_to_ppgtt((vm))); + + return err; +} + +static void __context_unpin_ppgtt(struct intel_context *ce) +{ + struct i915_address_space *vm; + + vm = vm_alias(ce->vm); + if (vm) + gen6_ppgtt_unpin(i915_vm_to_ppgtt(vm)); +} + static void ring_context_unpin(struct intel_context *ce) { + __context_unpin_ppgtt(ce); } static int alloc_context_vma(struct intel_context *ce) @@ -593,7 +896,7 @@ static int ring_context_alloc(struct intel_context *ce) static int ring_context_pin(struct intel_context *ce) { - return 0; + return __context_pin_ppgtt(ce); } static void ring_context_reset(struct intel_context *ce) @@ -649,12 +952,19 @@ static void ring_release(struct intel_engine_cs *engine) set_current_context(&engine->legacy.context, NULL); + if (engine->wa_ctx.vma) { + intel_context_put(engine->wa_ctx.vma->private); + i915_vma_unpin_and_release(&engine->wa_ctx.vma, 0); + } + intel_ring_unpin(engine->legacy.ring); intel_ring_put(engine->legacy.ring); } static void setup_irq(struct intel_engine_cs *engine) { + engine->irq_enable = gen6_irq_enable; + engine->irq_disable = gen6_irq_disable; } static void setup_common(struct intel_engine_cs *engine) @@ -663,7 +973,7 @@ static void setup_common(struct intel_engine_cs *engine) /* gen8+ are only supported with execlists */ GEM_BUG_ON(INTEL_GEN(i915) >= 8); - GEM_BUG_ON(INTEL_GEN(i915) < 8); + GEM_BUG_ON(INTEL_GEN(i915) < 6); setup_irq(engine); @@ -680,24 +990,62 @@ static void setup_common(struct intel_engine_cs *engine) engine->cops = &ring_context_ops; engine->request_alloc = ring_request_alloc; + engine->emit_init_breadcrumb = gen6_emit_init_breadcrumb; + if (INTEL_GEN(i915) >= 7) + engine->emit_fini_breadcrumb = gen7_emit_breadcrumb_xcs; + else if (INTEL_GEN(i915) >= 6) + engine->emit_fini_breadcrumb = gen6_emit_breadcrumb_xcs; + else + engine->emit_fini_breadcrumb = gen3_emit_breadcrumb; + engine->set_default_submission = set_default_submission; + + engine->emit_bb_start = gen6_emit_bb_start; } static void setup_rcs(struct intel_engine_cs *engine) { + struct drm_i915_private *i915 = engine->i915; + + if (HAS_L3_DPF(i915)) + engine->irq_keep_mask = GT_RENDER_L3_PARITY_ERROR_INTERRUPT; + + engine->irq_enable_mask = GT_RENDER_USER_INTERRUPT; + + if (INTEL_GEN(i915) >= 7) { + engine->emit_flush = gen7_emit_flush_rcs; + engine->emit_fini_breadcrumb = gen7_emit_breadcrumb_rcs; + if (IS_HASWELL(i915)) + engine->emit_bb_start = hsw_emit_bb_start; + } else { + engine->emit_flush = gen6_emit_flush_rcs; + engine->emit_fini_breadcrumb = gen6_emit_breadcrumb_rcs; + } } static void setup_vcs(struct intel_engine_cs *engine) { + engine->emit_flush = gen6_emit_flush_vcs; + engine->irq_enable_mask = GT_BSD_USER_INTERRUPT; + + if (IS_GEN(engine->i915, 6)) + engine->fw_domain = FORCEWAKE_ALL; } static void setup_bcs(struct intel_engine_cs *engine) { + engine->emit_flush = gen6_emit_flush_xcs; + engine->irq_enable_mask = GT_BLT_USER_INTERRUPT; } static void setup_vecs(struct intel_engine_cs *engine) { GEM_BUG_ON(!IS_HASWELL(engine->i915)); + + engine->emit_flush = gen6_emit_flush_xcs; + engine->irq_enable_mask = PM_VEBOX_USER_INTERRUPT; + engine->irq_enable = hsw_irq_enable_vecs; + engine->irq_disable = hsw_irq_disable_vecs; } static unsigned int global_ring_size(void) @@ -706,6 +1054,58 @@ static unsigned int global_ring_size(void) return roundup_pow_of_two(EXECLIST_MAX_PORTS * SZ_16K + SZ_4K); } +static int gen7_ctx_switch_bb_init(struct intel_engine_cs *engine) +{ + struct drm_i915_gem_object *obj; + struct i915_vma *vma; + int size; + int err; + + size = gen7_setup_clear_gpr_bb(engine, NULL /* probe size */); + if (size <= 0) + return size; + + size = ALIGN(size, PAGE_SIZE); + obj = i915_gem_object_create_internal(engine->i915, size); + if (IS_ERR(obj)) + return PTR_ERR(obj); + + vma = i915_vma_instance(obj, engine->gt->vm, NULL); + if (IS_ERR(vma)) { + err = PTR_ERR(vma); + goto err_obj; + } + + vma->private = intel_context_create(engine); /* dummy residuals */ + if (IS_ERR(vma->private)) { + err = PTR_ERR(vma->private); + goto err_obj; + } + + err = i915_vma_pin(vma, 0, 0, PIN_USER | PIN_HIGH); + if (err) + goto err_private; + + err = i915_vma_sync(vma); + if (err) + goto err_unpin; + + size = gen7_setup_clear_gpr_bb(engine, vma); + if (err) + goto err_unpin; + + engine->wa_ctx.vma = vma; + return 0; + +err_unpin: + i915_vma_unpin(vma); +err_private: + intel_context_put(vma->private); +err_obj: + i915_gem_object_put(obj); + return err; +} + int intel_ring_scheduler_setup(struct intel_engine_cs *engine) { struct intel_ring *ring; @@ -749,6 +1149,12 @@ int intel_ring_scheduler_setup(struct intel_engine_cs *engine) GEM_BUG_ON(engine->legacy.ring); engine->legacy.ring = ring; + if (IS_HASWELL(engine->i915) && engine->class == RENDER_CLASS) { + err = gen7_ctx_switch_bb_init(engine); + if (err) + goto err_ring_unpin; + } + engine->flags |= I915_ENGINE_NEEDS_BREADCRUMB_TASKLET; engine->flags |= I915_ENGINE_SUPPORTS_STATS; @@ -756,6 +1162,8 @@ int intel_ring_scheduler_setup(struct intel_engine_cs *engine) engine->release = ring_release; return 0; +err_ring_unpin: + intel_ring_unpin(ring); err_ring: intel_ring_put(ring); err: From patchwork Fri Jun 5 12:23:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 11589569 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F8EC912 for ; Fri, 5 Jun 2020 12:23:57 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4829B206E6 for ; Fri, 5 Jun 2020 12:23:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4829B206E6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=chris-wilson.co.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id EB7DE6E849; Fri, 5 Jun 2020 12:23:51 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id EE4C16E0A5 for ; Fri, 5 Jun 2020 12:23:47 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from build.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 21406198-1500050 for multiple; Fri, 05 Jun 2020 13:23:39 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Fri, 5 Jun 2020 13:23:34 +0100 Message-Id: <20200605122334.2798-10-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200605122334.2798-1-chris@chris-wilson.co.uk> References: <20200605122334.2798-1-chris@chris-wilson.co.uk> MIME-Version: 1.0 Subject: [Intel-gfx] [PATCH 10/10] drm/i915/gt: Enable ring scheduling for gen6/7 X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chris Wilson Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" Switch over from FIFO global submission to the priority-sorted topographical scheduler. At the cost of more busy work on the CPU to keep the GPU supplied with the next packet of requests, this allows us to reorder requests around submission stalls. This also enables the timer based RPS, with the exception of Valleyview who's PCU doesn't take kindly to our interference. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c | 2 +- drivers/gpu/drm/i915/gt/intel_engine_cs.c | 2 ++ drivers/gpu/drm/i915/gt/intel_rps.c | 6 ++---- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c index b81978890641..bb57687aea99 100644 --- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c +++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_context.c @@ -94,7 +94,7 @@ static int live_nop_switch(void *arg) rq = i915_request_get(this); i915_request_add(this); } - if (i915_request_wait(rq, 0, HZ / 5) < 0) { + if (i915_request_wait(rq, 0, HZ) < 0) { pr_err("Failed to populated %d contexts\n", nctx); intel_gt_set_wedged(&i915->gt); i915_request_put(rq); diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c index 4b36378af119..2312e8313325 100644 --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c @@ -790,6 +790,8 @@ int intel_engines_init(struct intel_gt *gt) if (HAS_EXECLISTS(gt->i915)) setup = intel_execlists_submission_setup; + else if (INTEL_GEN(gt->i915) >= 6) + setup = intel_ring_scheduler_setup; else setup = intel_ring_submission_setup; diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c index 2e4ddc9ca09d..22882c2953da 100644 --- a/drivers/gpu/drm/i915/gt/intel_rps.c +++ b/drivers/gpu/drm/i915/gt/intel_rps.c @@ -1053,9 +1053,7 @@ static bool gen6_rps_enable(struct intel_rps *rps) intel_uncore_write_fw(uncore, GEN6_RP_DOWN_TIMEOUT, 50000); intel_uncore_write_fw(uncore, GEN6_RP_IDLE_HYSTERSIS, 10); - rps->pm_events = (GEN6_PM_RP_UP_THRESHOLD | - GEN6_PM_RP_DOWN_THRESHOLD | - GEN6_PM_RP_DOWN_TIMEOUT); + rps->pm_events = GEN6_PM_RP_UP_THRESHOLD | GEN6_PM_RP_DOWN_THRESHOLD; return rps_reset(rps); } @@ -1362,7 +1360,7 @@ void intel_rps_enable(struct intel_rps *rps) GEM_BUG_ON(rps->efficient_freq < rps->min_freq); GEM_BUG_ON(rps->efficient_freq > rps->max_freq); - if (has_busy_stats(rps)) + if (has_busy_stats(rps) && !IS_VALLEYVIEW(i915)) intel_rps_set_timer(rps); else if (INTEL_GEN(i915) >= 6) intel_rps_set_interrupts(rps);