[v3] drm/i915: Move CSB MMIO reads out of the execlists lock
diff mbox

Message ID 1458219586-20452-1-git-send-email-tvrtko.ursulin@linux.intel.com
State New
Headers show

Commit Message

Tvrtko Ursulin March 17, 2016, 12:59 p.m. UTC
From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

By reading the CSB (slow MMIO accesses) into a temporary local
buffer we can decrease the duration of holding the execlist
lock.

Main advantage is that during heavy batch buffer submission we
reduce the execlist lock contention, which should decrease the
latency and CPU usage between the submitting userspace process
and interrupt handling.

Downside is that we need to grab and relase the forcewake twice,
but as the below numbers will show this is completely hidden
by the primary gains.

Testing with "gem_latency -n 100" (submit batch buffers with a
hundred nops each) shows more than doubling of the throughput
and more than halving of the dispatch latency, overall latency
and CPU time spend in the submitting process.

Submitting empty batches ("gem_latency -n 0") does not seem
significantly affected by this change with throughput and CPU
time improving by half a percent, and overall latency worsening
by the same amount.

Above tests were done in a hundred runs on a big core Broadwell.

v2:
  * Overflow protection to local CSB buffer.
  * Use closer dev_priv in execlists_submit_requests. (Chris Wilson)

v3: Rebase.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
---
 drivers/gpu/drm/i915/intel_lrc.c | 77 ++++++++++++++++++++--------------------
 1 file changed, 38 insertions(+), 39 deletions(-)

Comments

Chris Wilson March 17, 2016, 1:14 p.m. UTC | #1
On Thu, Mar 17, 2016 at 12:59:46PM +0000, Tvrtko Ursulin wrote:
> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> 
> By reading the CSB (slow MMIO accesses) into a temporary local
> buffer we can decrease the duration of holding the execlist
> lock.
> 
> Main advantage is that during heavy batch buffer submission we
> reduce the execlist lock contention, which should decrease the
> latency and CPU usage between the submitting userspace process
> and interrupt handling.
> 
> Downside is that we need to grab and relase the forcewake twice,
> but as the below numbers will show this is completely hidden
> by the primary gains.
> 
> Testing with "gem_latency -n 100" (submit batch buffers with a
> hundred nops each) shows more than doubling of the throughput
> and more than halving of the dispatch latency, overall latency
> and CPU time spend in the submitting process.
> 
> Submitting empty batches ("gem_latency -n 0") does not seem
> significantly affected by this change with throughput and CPU
> time improving by half a percent, and overall latency worsening
> by the same amount.
> 
> Above tests were done in a hundred runs on a big core Broadwell.
> 
> v2:
>   * Overflow protection to local CSB buffer.
>   * Use closer dev_priv in execlists_submit_requests. (Chris Wilson)
> 
> v3: Rebase.
> 
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> Cc: Chris Wilson <chris@chris-wilson.co.uk>
> ---
>  drivers/gpu/drm/i915/intel_lrc.c | 77 ++++++++++++++++++++--------------------
>  1 file changed, 38 insertions(+), 39 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
> index f72782200226..c6b3a9d19af2 100644
> --- a/drivers/gpu/drm/i915/intel_lrc.c
> +++ b/drivers/gpu/drm/i915/intel_lrc.c
> @@ -416,15 +416,23 @@ static void execlists_update_context(struct drm_i915_gem_request *rq)
>  static void execlists_submit_requests(struct drm_i915_gem_request *rq0,
>  				      struct drm_i915_gem_request *rq1)
>  {
> +	struct drm_i915_private *dev_priv = rq0->i915;
> +
>  	execlists_update_context(rq0);
>  
>  	if (rq1)
>  		execlists_update_context(rq1);
>  
> +	spin_lock(&dev_priv->uncore.lock);
> +	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);

My only problem with this is that it puts the onus on the caller to
remember to disable irqs first. (That would be a silent conflict with
the patch to move the submission to a kthread for example.)
What's the cost of being upfront with spin_lock_irqsave() here?
/* BUG_ON(!irqs_disabled());  */ ?

I'm quite happy to go with the comment here and since it doesn't make
the lockups any worse,
Reviewed-by: Chris Wilsno <chris@chris-wilson.co.uk>
-Chris
Tvrtko Ursulin March 17, 2016, 4:30 p.m. UTC | #2
On 17/03/16 13:14, Chris Wilson wrote:
> On Thu, Mar 17, 2016 at 12:59:46PM +0000, Tvrtko Ursulin wrote:
>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>
>> By reading the CSB (slow MMIO accesses) into a temporary local
>> buffer we can decrease the duration of holding the execlist
>> lock.
>>
>> Main advantage is that during heavy batch buffer submission we
>> reduce the execlist lock contention, which should decrease the
>> latency and CPU usage between the submitting userspace process
>> and interrupt handling.
>>
>> Downside is that we need to grab and relase the forcewake twice,
>> but as the below numbers will show this is completely hidden
>> by the primary gains.
>>
>> Testing with "gem_latency -n 100" (submit batch buffers with a
>> hundred nops each) shows more than doubling of the throughput
>> and more than halving of the dispatch latency, overall latency
>> and CPU time spend in the submitting process.
>>
>> Submitting empty batches ("gem_latency -n 0") does not seem
>> significantly affected by this change with throughput and CPU
>> time improving by half a percent, and overall latency worsening
>> by the same amount.
>>
>> Above tests were done in a hundred runs on a big core Broadwell.
>>
>> v2:
>>    * Overflow protection to local CSB buffer.
>>    * Use closer dev_priv in execlists_submit_requests. (Chris Wilson)
>>
>> v3: Rebase.
>>
>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>> Cc: Chris Wilson <chris@chris-wilson.co.uk>
>> ---
>>   drivers/gpu/drm/i915/intel_lrc.c | 77 ++++++++++++++++++++--------------------
>>   1 file changed, 38 insertions(+), 39 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
>> index f72782200226..c6b3a9d19af2 100644
>> --- a/drivers/gpu/drm/i915/intel_lrc.c
>> +++ b/drivers/gpu/drm/i915/intel_lrc.c
>> @@ -416,15 +416,23 @@ static void execlists_update_context(struct drm_i915_gem_request *rq)
>>   static void execlists_submit_requests(struct drm_i915_gem_request *rq0,
>>   				      struct drm_i915_gem_request *rq1)
>>   {
>> +	struct drm_i915_private *dev_priv = rq0->i915;
>> +
>>   	execlists_update_context(rq0);
>>
>>   	if (rq1)
>>   		execlists_update_context(rq1);
>>
>> +	spin_lock(&dev_priv->uncore.lock);
>> +	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
>
> My only problem with this is that it puts the onus on the caller to
> remember to disable irqs first. (That would be a silent conflict with
> the patch to move the submission to a kthread for example.)

Oh well in this code ones has to know what they are doing anyway so I 
consider this very minor.

> What's the cost of being upfront with spin_lock_irqsave() here?

No idea but probably not much if at all detectable. Issue I have with 
that that elsewhere we have this approach of using the exact right one 
for the use case.

> /* BUG_ON(!irqs_disabled());  */ ?

As a comment?

> I'm quite happy to go with the comment here and since it doesn't make
> the lockups any worse,
> Reviewed-by: Chris Wilsno <chris@chris-wilson.co.uk>

Thanks, will add that comment.

Regards,

Tvrtko
Tvrtko Ursulin March 18, 2016, 10:31 a.m. UTC | #3
On 18/03/16 07:02, Patchwork wrote:
> == Series Details ==
>
> Series: drm/i915: Move CSB MMIO reads out of the execlists lock (rev4)
> URL   : https://patchwork.freedesktop.org/series/3973/
> State : failure
>
> == Summary ==
>
> Series 3973v4 drm/i915: Move CSB MMIO reads out of the execlists lock
> http://patchwork.freedesktop.org/api/1.0/series/3973/revisions/4/mbox/
>
> Test gem_ringfill:
>          Subgroup basic-default-s3:
>                  dmesg-warn -> PASS       (skl-nuci5)
> Test gem_storedw_loop:
>          Subgroup basic-default:
>                  dmesg-warn -> PASS       (skl-nuci5)
> Test gem_sync:
>          Subgroup basic-vebox:
>                  dmesg-warn -> PASS       (skl-nuci5)
> Test kms_flip:
>          Subgroup basic-flip-vs-wf_vblank:
>                  pass       -> DMESG-WARN (bdw-ultra)

Device suspended during HW access: 
https://bugs.freedesktop.org/show_bug.cgi?id=94349

> Test kms_pipe_crc_basic:
>          Subgroup nonblocking-crc-pipe-c-frame-sequence:
>                  pass       -> DMESG-WARN (hsw-brixbox)

Unclaimed register: https://bugs.freedesktop.org/show_bug.cgi?id=94384

>          Subgroup read-crc-pipe-a-frame-sequence:
>                  dmesg-warn -> PASS       (hsw-gt2)
>          Subgroup suspend-read-crc-pipe-b:
>                  pass       -> INCOMPLETE (hsw-gt2)

Haswell does not use the code touched here so I am discounting this as 
unrelated.

>          Subgroup suspend-read-crc-pipe-c:
>                  dmesg-warn -> PASS       (bsw-nuc-2)
> Test pm_rpm:
>          Subgroup basic-pci-d3-state:
>                  pass       -> DMESG-WARN (hsw-brixbox)

Device suspended during HW access: 
https://bugs.freedesktop.org/show_bug.cgi?id=94349

>
> bdw-ultra        total:194  pass:171  dwarn:2   dfail:0   fail:0   skip:21
> bsw-nuc-2        total:194  pass:156  dwarn:1   dfail:0   fail:0   skip:37
> hsw-brixbox      total:194  pass:169  dwarn:3   dfail:0   fail:0   skip:22
> hsw-gt2          total:115  pass:104  dwarn:1   dfail:0   fail:0   skip:9
> ivb-t430s        total:194  pass:168  dwarn:1   dfail:0   fail:0   skip:25
> skl-i5k-2        total:194  pass:170  dwarn:1   dfail:0   fail:0   skip:23
> skl-i7k-2        total:194  pass:170  dwarn:1   dfail:0   fail:0   skip:23
> skl-nuci5        total:194  pass:182  dwarn:1   dfail:0   fail:0   skip:11
>
> Results at /archive/results/CI_IGT_test/Patchwork_1632/
>
> 10e913a48ca36790da9b58bed8729598ea79ebdb drm-intel-nightly: 2016y-03m-17d-13h-22m-41s UTC integration manifest
> f6a228bfa750f3f23c1a95ca76f08b148a02f2a5 drm/i915: Move CSB MMIO reads out of the execlists lock

Merged, thanks for the review.

Regards,

Tvrtko
Daniel Vetter March 21, 2016, 9:22 a.m. UTC | #4
On Thu, Mar 17, 2016 at 04:30:47PM +0000, Tvrtko Ursulin wrote:
> 
> On 17/03/16 13:14, Chris Wilson wrote:
> >On Thu, Mar 17, 2016 at 12:59:46PM +0000, Tvrtko Ursulin wrote:
> >>From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> >>
> >>By reading the CSB (slow MMIO accesses) into a temporary local
> >>buffer we can decrease the duration of holding the execlist
> >>lock.
> >>
> >>Main advantage is that during heavy batch buffer submission we
> >>reduce the execlist lock contention, which should decrease the
> >>latency and CPU usage between the submitting userspace process
> >>and interrupt handling.
> >>
> >>Downside is that we need to grab and relase the forcewake twice,
> >>but as the below numbers will show this is completely hidden
> >>by the primary gains.
> >>
> >>Testing with "gem_latency -n 100" (submit batch buffers with a
> >>hundred nops each) shows more than doubling of the throughput
> >>and more than halving of the dispatch latency, overall latency
> >>and CPU time spend in the submitting process.
> >>
> >>Submitting empty batches ("gem_latency -n 0") does not seem
> >>significantly affected by this change with throughput and CPU
> >>time improving by half a percent, and overall latency worsening
> >>by the same amount.
> >>
> >>Above tests were done in a hundred runs on a big core Broadwell.
> >>
> >>v2:
> >>   * Overflow protection to local CSB buffer.
> >>   * Use closer dev_priv in execlists_submit_requests. (Chris Wilson)
> >>
> >>v3: Rebase.
> >>
> >>Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> >>Cc: Chris Wilson <chris@chris-wilson.co.uk>
> >>---
> >>  drivers/gpu/drm/i915/intel_lrc.c | 77 ++++++++++++++++++++--------------------
> >>  1 file changed, 38 insertions(+), 39 deletions(-)
> >>
> >>diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
> >>index f72782200226..c6b3a9d19af2 100644
> >>--- a/drivers/gpu/drm/i915/intel_lrc.c
> >>+++ b/drivers/gpu/drm/i915/intel_lrc.c
> >>@@ -416,15 +416,23 @@ static void execlists_update_context(struct drm_i915_gem_request *rq)
> >>  static void execlists_submit_requests(struct drm_i915_gem_request *rq0,
> >>  				      struct drm_i915_gem_request *rq1)
> >>  {
> >>+	struct drm_i915_private *dev_priv = rq0->i915;
> >>+
> >>  	execlists_update_context(rq0);
> >>
> >>  	if (rq1)
> >>  		execlists_update_context(rq1);
> >>
> >>+	spin_lock(&dev_priv->uncore.lock);
> >>+	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
> >
> >My only problem with this is that it puts the onus on the caller to
> >remember to disable irqs first. (That would be a silent conflict with
> >the patch to move the submission to a kthread for example.)
> 
> Oh well in this code ones has to know what they are doing anyway so I
> consider this very minor.
> 
> >What's the cost of being upfront with spin_lock_irqsave() here?
> 
> No idea but probably not much if at all detectable. Issue I have with that
> that elsewhere we have this approach of using the exact right one for the
> use case.
> 
> >/* BUG_ON(!irqs_disabled());  */ ?
> 
> As a comment?

As a line of code. I agree with Chris that we should add this, making it
even harder for newbies to dig into gem/execlist code doesn't help anyone.
With this line we get up to

"Get it right or it breaks at runtime."

Before that we are at

"Read the right mailing list thread and you get it right"

Per http://sweng.the-davies.net/Home/rustys-api-design-manifesto

Ofc if it has significant overhead then we need to reconsider.
-Daniel

> 
> >I'm quite happy to go with the comment here and since it doesn't make
> >the lockups any worse,
> >Reviewed-by: Chris Wilsno <chris@chris-wilson.co.uk>
> 
> Thanks, will add that comment.
> 
> Regards,
> 
> Tvrtko
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Patch
diff mbox

diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index f72782200226..c6b3a9d19af2 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -416,15 +416,23 @@  static void execlists_update_context(struct drm_i915_gem_request *rq)
 static void execlists_submit_requests(struct drm_i915_gem_request *rq0,
 				      struct drm_i915_gem_request *rq1)
 {
+	struct drm_i915_private *dev_priv = rq0->i915;
+
 	execlists_update_context(rq0);
 
 	if (rq1)
 		execlists_update_context(rq1);
 
+	spin_lock(&dev_priv->uncore.lock);
+	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
+
 	execlists_elsp_write(rq0, rq1);
+
+	intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL);
+	spin_unlock(&dev_priv->uncore.lock);
 }
 
-static void execlists_context_unqueue__locked(struct intel_engine_cs *engine)
+static void execlists_context_unqueue(struct intel_engine_cs *engine)
 {
 	struct drm_i915_gem_request *req0 = NULL, *req1 = NULL;
 	struct drm_i915_gem_request *cursor, *tmp;
@@ -478,19 +486,6 @@  static void execlists_context_unqueue__locked(struct intel_engine_cs *engine)
 	execlists_submit_requests(req0, req1);
 }
 
-static void execlists_context_unqueue(struct intel_engine_cs *engine)
-{
-	struct drm_i915_private *dev_priv = engine->dev->dev_private;
-
-	spin_lock(&dev_priv->uncore.lock);
-	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
-
-	execlists_context_unqueue__locked(engine);
-
-	intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL);
-	spin_unlock(&dev_priv->uncore.lock);
-}
-
 static unsigned int
 execlists_check_remove_request(struct intel_engine_cs *engine, u32 request_id)
 {
@@ -551,12 +546,10 @@  void intel_lrc_irq_handler(struct intel_engine_cs *engine)
 	struct drm_i915_private *dev_priv = engine->dev->dev_private;
 	u32 status_pointer;
 	unsigned int read_pointer, write_pointer;
-	u32 status = 0;
-	u32 status_id;
+	u32 csb[GEN8_CSB_ENTRIES][2];
+	unsigned int csb_read = 0, i;
 	unsigned int submit_contexts = 0;
 
-	spin_lock(&engine->execlist_lock);
-
 	spin_lock(&dev_priv->uncore.lock);
 	intel_uncore_forcewake_get__locked(dev_priv, FORCEWAKE_ALL);
 
@@ -568,41 +561,47 @@  void intel_lrc_irq_handler(struct intel_engine_cs *engine)
 		write_pointer += GEN8_CSB_ENTRIES;
 
 	while (read_pointer < write_pointer) {
-		status = get_context_status(engine, ++read_pointer,
-				            &status_id);
+		if (WARN_ON_ONCE(csb_read == GEN8_CSB_ENTRIES))
+			break;
+		csb[csb_read][0] = get_context_status(engine, ++read_pointer,
+						      &csb[csb_read][1]);
+		csb_read++;
+	}
 
-		if (unlikely(status & GEN8_CTX_STATUS_PREEMPTED)) {
-			if (status & GEN8_CTX_STATUS_LITE_RESTORE) {
-				if (execlists_check_remove_request(engine, status_id))
+	engine->next_context_status_buffer = write_pointer % GEN8_CSB_ENTRIES;
+
+	/* Update the read pointer to the old write pointer. Manual ringbuffer
+	 * management ftw </sarcasm> */
+	I915_WRITE_FW(RING_CONTEXT_STATUS_PTR(engine),
+		      _MASKED_FIELD(GEN8_CSB_READ_PTR_MASK,
+				    engine->next_context_status_buffer << 8));
+
+	intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL);
+	spin_unlock(&dev_priv->uncore.lock);
+
+	spin_lock(&engine->execlist_lock);
+
+	for (i = 0; i < csb_read; i++) {
+		if (unlikely(csb[i][0] & GEN8_CTX_STATUS_PREEMPTED)) {
+			if (csb[i][0] & GEN8_CTX_STATUS_LITE_RESTORE) {
+				if (execlists_check_remove_request(engine, csb[i][1]))
 					WARN(1, "Lite Restored request removed from queue\n");
 			} else
 				WARN(1, "Preemption without Lite Restore\n");
 		}
 
-		if (status & (GEN8_CTX_STATUS_ACTIVE_IDLE |
+		if (csb[i][0] & (GEN8_CTX_STATUS_ACTIVE_IDLE |
 		    GEN8_CTX_STATUS_ELEMENT_SWITCH))
 			submit_contexts +=
-				execlists_check_remove_request(engine,
-						               status_id);
+				execlists_check_remove_request(engine, csb[i][1]);
 	}
 
 	if (submit_contexts) {
 		if (!engine->disable_lite_restore_wa ||
-		    (status & GEN8_CTX_STATUS_ACTIVE_IDLE))
-			execlists_context_unqueue__locked(engine);
+		    (csb[i][0] & GEN8_CTX_STATUS_ACTIVE_IDLE))
+			execlists_context_unqueue(engine);
 	}
 
-	engine->next_context_status_buffer = write_pointer % GEN8_CSB_ENTRIES;
-
-	/* Update the read pointer to the old write pointer. Manual ringbuffer
-	 * management ftw </sarcasm> */
-	I915_WRITE_FW(RING_CONTEXT_STATUS_PTR(engine),
-		      _MASKED_FIELD(GEN8_CSB_READ_PTR_MASK,
-				    engine->next_context_status_buffer << 8));
-
-	intel_uncore_forcewake_put__locked(dev_priv, FORCEWAKE_ALL);
-	spin_unlock(&dev_priv->uncore.lock);
-
 	spin_unlock(&engine->execlist_lock);
 
 	if (unlikely(submit_contexts > 2))