From patchwork Thu Feb 9 15:43:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 13134793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 20EFBC636D6 for ; Thu, 9 Feb 2023 15:43:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 455D510EB31; Thu, 9 Feb 2023 15:43:33 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7A00E10EB2E; Thu, 9 Feb 2023 15:43:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675957410; x=1707493410; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lrPdTfl1YDJNIs0dJwyIhJGh23rQm1EcY6zS5LmHiGo=; b=n4Ra9qELHME0fB1RmKlxjmOpAapmXtGAXVdt4/eXK695vcIY7s9lJ/Kv P9pGB5vs1eIhv9Fdzze3+qA+Lv0KssHUmjnhohBsCv3MSc5vr3MgDBV4q Tibx3WXcpzPePjMppGNR40Aw1fF3GxMmSEkrtriqI1pdISE33gzpbfg/k 83wA14BLKtpJ8CdS9NlCLi11RHNkeI2K2S+O3m6vrIL3Ncteb8lMGa4tO jumxnjvgaGxOBrfny995t+mFwYFjzu67rc0ajNRtxUEeP93EnlQAfAldw mEU0h6bQIEfyr/J9S7b7ZnD/Un9VUSGlghVli+2IzADUonqzjh4oojB18 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="331440708" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="331440708" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:30 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="645308790" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="645308790" Received: from jptoner-mobl2.ger.corp.intel.com (HELO localhost.localdomain) ([10.213.192.123]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:28 -0800 From: Tvrtko Ursulin To: Intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Thu, 9 Feb 2023 15:43:13 +0000 Message-Id: <20230209154315.468682-2-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> References: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [RFC 1/3] dma-fence: Track explicit waiters X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" From: Tvrtko Ursulin Track how many callers are explicity waiting on a fence to signal and allow querying that via new dma_fence_wait_count() API. This provides infrastructure on top of which generic "waitboost" concepts can be implemented by individual drivers. Wait-boosting is any reactive activity, such as raising the GPU clocks, which happens while there are active external waiters. Signed-off-by: Tvrtko Ursulin --- drivers/dma-buf/dma-fence.c | 102 ++++++++++++++++++++++++++---------- include/linux/dma-fence.h | 14 +++++ 2 files changed, 88 insertions(+), 28 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index 0de0482cd36e..d463c06ebd6d 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -344,6 +344,24 @@ void __dma_fence_might_wait(void) } #endif +static void incr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb) +{ + lockdep_assert_held(fence->lock); + + __set_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags); + fence->waitcount++; + WARN_ON_ONCE(!fence->waitcount); +} + +static void decr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb) +{ + lockdep_assert_held(fence->lock); + + if (__test_and_clear_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags)) { + WARN_ON_ONCE(!fence->waitcount); + fence->waitcount--; + } +} /** * dma_fence_signal_timestamp_locked - signal completion of a fence @@ -384,6 +402,7 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence, list_for_each_entry_safe(cur, tmp, &cb_list, node) { INIT_LIST_HEAD(&cur->node); + decr_wait_count(fence, cur); cur->func(fence, cur); } @@ -612,35 +631,15 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence) unsigned long flags; spin_lock_irqsave(fence->lock, flags); + fence->waitcount++; + WARN_ON_ONCE(!fence->waitcount); __dma_fence_enable_signaling(fence); spin_unlock_irqrestore(fence->lock, flags); } EXPORT_SYMBOL(dma_fence_enable_sw_signaling); -/** - * dma_fence_add_callback - add a callback to be called when the fence - * is signaled - * @fence: the fence to wait on - * @cb: the callback to register - * @func: the function to call - * - * Add a software callback to the fence. The caller should keep a reference to - * the fence. - * - * @cb will be initialized by dma_fence_add_callback(), no initialization - * by the caller is required. Any number of callbacks can be registered - * to a fence, but a callback can only be registered to one fence at a time. - * - * If fence is already signaled, this function will return -ENOENT (and - * *not* call the callback). - * - * Note that the callback can be called from an atomic context or irq context. - * - * Returns 0 in case of success, -ENOENT if the fence is already signaled - * and -EINVAL in case of error. - */ -int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, - dma_fence_func_t func) +static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, + dma_fence_func_t func, bool wait) { unsigned long flags; int ret = 0; @@ -655,10 +654,15 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, spin_lock_irqsave(fence->lock, flags); + if (wait) + incr_wait_count(fence, cb); + if (__dma_fence_enable_signaling(fence)) { cb->func = func; list_add_tail(&cb->node, &fence->cb_list); } else { + if (wait) + decr_wait_count(fence, cb); INIT_LIST_HEAD(&cb->node); ret = -ENOENT; } @@ -667,8 +671,44 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, return ret; } + +/** + * dma_fence_add_callback - add a callback to be called when the fence + * is signaled + * @fence: the fence to wait on + * @cb: the callback to register + * @func: the function to call + * + * Add a software callback to the fence. The caller should keep a reference to + * the fence. + * + * @cb will be initialized by dma_fence_add_callback(), no initialization + * by the caller is required. Any number of callbacks can be registered + * to a fence, but a callback can only be registered to one fence at a time. + * + * If fence is already signaled, this function will return -ENOENT (and + * *not* call the callback). + * + * Note that the callback can be called from an atomic context or irq context. + * + * Returns 0 in case of success, -ENOENT if the fence is already signaled + * and -EINVAL in case of error. + */ +int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, + dma_fence_func_t func) +{ + return add_callback(fence, cb, func, false); +} EXPORT_SYMBOL(dma_fence_add_callback); +int dma_fence_add_wait_callback(struct dma_fence *fence, + struct dma_fence_cb *cb, + dma_fence_func_t func) +{ + return add_callback(fence, cb, func, true); +} +EXPORT_SYMBOL(dma_fence_add_wait_callback); + /** * dma_fence_get_status - returns the status upon completion * @fence: the dma_fence to query @@ -721,8 +761,10 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb) spin_lock_irqsave(fence->lock, flags); ret = !list_empty(&cb->node); - if (ret) + if (ret) { + decr_wait_count(fence, cb); list_del_init(&cb->node); + } spin_unlock_irqrestore(fence->lock, flags); @@ -780,6 +822,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) cb.base.func = dma_fence_default_wait_cb; cb.task = current; + incr_wait_count(fence, &cb.base); list_add(&cb.base.node, &fence->cb_list); while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) { @@ -796,8 +839,10 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) ret = -ERESTARTSYS; } - if (!list_empty(&cb.base.node)) + if (!list_empty(&cb.base.node)) { + decr_wait_count(fence, &cb.base); list_del(&cb.base.node); + } __set_current_state(TASK_RUNNING); out: @@ -875,8 +920,8 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count, struct dma_fence *fence = fences[i]; cb[i].task = current; - if (dma_fence_add_callback(fence, &cb[i].base, - dma_fence_default_wait_cb)) { + if (dma_fence_add_wait_callback(fence, &cb[i].base, + dma_fence_default_wait_cb)) { /* This fence is already signaled */ if (idx) *idx = i; @@ -957,6 +1002,7 @@ dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, fence->context = context; fence->seqno = seqno; fence->flags = 0UL; + fence->waitcount = 0; fence->error = 0; trace_dma_fence_init(fence); diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index 775cdc0b4f24..d0ed923e4545 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -92,6 +92,7 @@ struct dma_fence { u64 seqno; unsigned long flags; struct kref refcount; + unsigned int waitcount; int error; }; @@ -116,6 +117,11 @@ typedef void (*dma_fence_func_t)(struct dma_fence *fence, struct dma_fence_cb { struct list_head node; dma_fence_func_t func; + unsigned long flags; +}; + +enum dma_fence_cb_flag_bits { + DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, }; /** @@ -377,6 +383,9 @@ signed long dma_fence_default_wait(struct dma_fence *fence, int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, dma_fence_func_t func); +int dma_fence_add_wait_callback(struct dma_fence *fence, + struct dma_fence_cb *cb, + dma_fence_func_t func); bool dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb); void dma_fence_enable_sw_signaling(struct dma_fence *fence); @@ -528,6 +537,11 @@ static inline int dma_fence_get_status_locked(struct dma_fence *fence) int dma_fence_get_status(struct dma_fence *fence); +static inline unsigned int dma_fence_wait_count(struct dma_fence *fence) +{ + return fence->waitcount; +} + /** * dma_fence_set_error - flag an error condition on the fence * @fence: the dma_fence From patchwork Thu Feb 9 15:43:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 13134795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2CED8C636D6 for ; Thu, 9 Feb 2023 15:43:48 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 620E310EB35; Thu, 9 Feb 2023 15:43:38 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id DF19310EB34; Thu, 9 Feb 2023 15:43:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675957411; x=1707493411; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zD70VZdjgnOic3K1qhG8DBPtshCjC6Dxxa8vF3F1mg4=; b=cvMhQQdzGzNghwV9Cd7QShYVs1ozihNpG8bPuTOF88czroE1IdKbjE/c /fuyYeQ19dyE8jSkmj7aZuaDXaFOIpWSQ0b8NtcZz29+e3ZSsw8lS5+Rd 1E8P0pjJzJ5pjmBXX0MSfInyQjN3ZPl0v+VhusC7oz0ez51pcXqRCv9l6 j4pztP4y4sfnN1oKxk25YPCCe/zEtROkVpDU+EmfSEmO+3XpS5/EzkGK8 5r9hHeg956x/2Bjm6IOa5ULAB6zJXJz2w0BW9php4MdTU1PF/XmtUUVDz 7whGTe3rZuKoFY0Gmezzl+sGQ64yhPhY7dg61m9jja9uQ/rW90i3aQsg8 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="331440713" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="331440713" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:31 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="645308796" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="645308796" Received: from jptoner-mobl2.ger.corp.intel.com (HELO localhost.localdomain) ([10.213.192.123]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:30 -0800 From: Tvrtko Ursulin To: Intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Thu, 9 Feb 2023 15:43:14 +0000 Message-Id: <20230209154315.468682-3-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> References: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [RFC 2/3] drm/syncobj: Mark syncobj waits as external waiters X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" From: Tvrtko Ursulin Use the previously added dma-fence tracking of explicit waiters. Signed-off-by: Tvrtko Ursulin --- drivers/gpu/drm/drm_syncobj.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/drm_syncobj.c b/drivers/gpu/drm/drm_syncobj.c index 0c2be8360525..776b90774a64 100644 --- a/drivers/gpu/drm/drm_syncobj.c +++ b/drivers/gpu/drm/drm_syncobj.c @@ -1065,9 +1065,9 @@ static signed long drm_syncobj_array_wait_timeout(struct drm_syncobj **syncobjs, if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) || dma_fence_is_signaled(fence) || (!entries[i].fence_cb.func && - dma_fence_add_callback(fence, - &entries[i].fence_cb, - syncobj_wait_fence_func))) { + dma_fence_add_wait_callback(fence, + &entries[i].fence_cb, + syncobj_wait_fence_func))) { /* The fence has been signaled */ if (flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL) { signaled_count++; From patchwork Thu Feb 9 15:43:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 13134794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8C625C61DA4 for ; Thu, 9 Feb 2023 15:43:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B4E9B10EB24; Thu, 9 Feb 2023 15:43:35 +0000 (UTC) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 69CCC10E213; Thu, 9 Feb 2023 15:43:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675957413; x=1707493413; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=WGRFM/BbKAw2CIFEGVHlyresbS5s3aUWJeKxqVMf1J0=; b=DwWSaljqo5ztV/EW8J6n7h+5Ob5F06GZsBsFmGWcfBdaeIXmP5clMwf3 GhDzxnUgbVSFJRuGAeOPRnV+UL4rdrxWrDkAsadinGn3IGTFjI281s0an CeZdCT1k0bMbQHU3VIzYQ1t7gA9SrOk0cEqAcyiwFmLfI2l/ScKOiKZUa aGM6aZXRhcKWhyI3iVilvikJg2qp+XDbhKFXb3SD0xZzkMQ+aFmOnEr/s L9jKMx72Bqaq1FmsRlWTULFz4dZm2O4lTfzTSYn8yE+WvtVSLIVAMU8A0 X0zBRdaNEhZDW9fNRREbL2mdfG3HF0Fkzxxi/rm7pF+w1m7oBEXzlTXTt g==; X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="331440718" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="331440718" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10615"; a="645308799" X-IronPort-AV: E=Sophos;i="5.97,284,1669104000"; d="scan'208";a="645308799" Received: from jptoner-mobl2.ger.corp.intel.com (HELO localhost.localdomain) ([10.213.192.123]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2023 07:43:31 -0800 From: Tvrtko Ursulin To: Intel-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Date: Thu, 9 Feb 2023 15:43:15 +0000 Message-Id: <20230209154315.468682-4-tvrtko.ursulin@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> References: <20230209154315.468682-1-tvrtko.ursulin@linux.intel.com> MIME-Version: 1.0 Subject: [Intel-gfx] [RFC 3/3] drm/i915: Waitboost external waits X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" From: Tvrtko Ursulin Userspace waits coming via the drm_syncobj route have so far been bypassing the waitboost mechanism. Use the previously added dma-fence wait tracking API and apply the same waitboosting logic which applies to other entry points. This should fix the perfomance regressions experience by clvk and similar userspace which relies on drm_syncobj. At the same time, but for documentation purposes only, use the new dma-fence API from i915_request_wait too. Signed-off-by: Tvrtko Ursulin --- drivers/gpu/drm/i915/i915_request.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c index 7503dcb9043b..e24fac5c1567 100644 --- a/drivers/gpu/drm/i915/i915_request.c +++ b/drivers/gpu/drm/i915/i915_request.c @@ -94,7 +94,12 @@ static bool i915_fence_signaled(struct dma_fence *fence) static bool i915_fence_enable_signaling(struct dma_fence *fence) { - return i915_request_enable_breadcrumb(to_request(fence)); + struct i915_request *rq = to_request(fence); + + if (dma_fence_wait_count(&rq->fence) && !i915_request_started(rq)) + intel_rps_boost(rq); + + return i915_request_enable_breadcrumb(rq); } static signed long i915_fence_wait(struct dma_fence *fence, @@ -2037,11 +2042,13 @@ long i915_request_wait_timeout(struct i915_request *rq, * but at a cost of spending more power processing the workload * (bad for battery). */ - if (flags & I915_WAIT_PRIORITY && !i915_request_started(rq)) + if (((flags & I915_WAIT_PRIORITY) || dma_fence_wait_count(&rq->fence)) + && !i915_request_started(rq)) intel_rps_boost(rq); wait.tsk = current; - if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake)) + if (dma_fence_add_wait_callback(&rq->fence, &wait.cb, + request_wait_wake)) goto out; /*