From patchwork Thu Aug 9 09:16:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 10561159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFA4B1515 for ; Thu, 9 Aug 2018 09:16:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C30B229CFA for ; Thu, 9 Aug 2018 09:16:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B7A462AB15; Thu, 9 Aug 2018 09:16:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5E4E229CFA for ; Thu, 9 Aug 2018 09:16:49 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9E26B6E077; Thu, 9 Aug 2018 09:16:48 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id A92F66E077 for ; Thu, 9 Aug 2018 09:16:44 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from haswell.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 12743769-1500050 for multiple; Thu, 09 Aug 2018 10:16:37 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Thu, 9 Aug 2018 10:16:34 +0100 Message-Id: <20180809091635.14976-1-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.18.0 Subject: [Intel-gfx] [PATCH 1/2] drm/i915: Introduce intel_runtime_pm_disable to pair intel_runtime_pm_enable X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP Currently, we cancel the extra wakeref we have for !runtime-pm devices inside power_wells_fini_hw. However, this is not strictly paired with the acquisition of that wakeref in runtime_pm_enable (as the fini_hw may be called on errors paths before we even call runtime_pm_enable). Make the symmetry more explicit and include a check that we do release all of our rpm wakerefs. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/i915_drv.c | 8 ++++++-- drivers/gpu/drm/i915/intel_drv.h | 1 + drivers/gpu/drm/i915/intel_runtime_pm.c | 24 +++++++++++++++--------- 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c index 9dce55182c3a..62ef105a241d 100644 --- a/drivers/gpu/drm/i915/i915_drv.c +++ b/drivers/gpu/drm/i915/i915_drv.c @@ -1281,6 +1281,8 @@ static void i915_driver_register(struct drm_i915_private *dev_priv) */ if (INTEL_INFO(dev_priv)->num_pipes) drm_kms_helper_poll_init(dev); + + intel_runtime_pm_enable(dev_priv); } /** @@ -1289,6 +1291,8 @@ static void i915_driver_register(struct drm_i915_private *dev_priv) */ static void i915_driver_unregister(struct drm_i915_private *dev_priv) { + intel_runtime_pm_disable(dev_priv); + intel_fbdev_unregister(dev_priv); intel_audio_deinit(dev_priv); @@ -1408,8 +1412,6 @@ int i915_driver_load(struct pci_dev *pdev, const struct pci_device_id *ent) i915_driver_register(dev_priv); - intel_runtime_pm_enable(dev_priv); - intel_init_ipc(dev_priv); intel_runtime_pm_put(dev_priv); @@ -1474,6 +1476,8 @@ void i915_driver_unload(struct drm_device *dev) i915_driver_cleanup_mmio(dev_priv); intel_display_power_put(dev_priv, POWER_DOMAIN_INIT); + + WARN_ON(atomic_read(&dev_priv->runtime_pm.wakeref_count)); } static void i915_driver_release(struct drm_device *dev) diff --git a/drivers/gpu/drm/i915/intel_drv.h b/drivers/gpu/drm/i915/intel_drv.h index 0601abb8c71f..dc6c0cec9b36 100644 --- a/drivers/gpu/drm/i915/intel_drv.h +++ b/drivers/gpu/drm/i915/intel_drv.h @@ -1956,6 +1956,7 @@ void intel_power_domains_verify_state(struct drm_i915_private *dev_priv); void bxt_display_core_init(struct drm_i915_private *dev_priv, bool resume); void bxt_display_core_uninit(struct drm_i915_private *dev_priv); void intel_runtime_pm_enable(struct drm_i915_private *dev_priv); +void intel_runtime_pm_disable(struct drm_i915_private *dev_priv); const char * intel_display_power_domain_str(enum intel_display_power_domain domain); diff --git a/drivers/gpu/drm/i915/intel_runtime_pm.c b/drivers/gpu/drm/i915/intel_runtime_pm.c index e209edbc561d..b78c3b48aa62 100644 --- a/drivers/gpu/drm/i915/intel_runtime_pm.c +++ b/drivers/gpu/drm/i915/intel_runtime_pm.c @@ -3793,8 +3793,6 @@ void intel_power_domains_init_hw(struct drm_i915_private *dev_priv, bool resume) */ void intel_power_domains_fini_hw(struct drm_i915_private *dev_priv) { - struct device *kdev = &dev_priv->drm.pdev->dev; - /* * The i915.ko module is still not prepared to be loaded when * the power well is not enabled, so just enable it in case @@ -3809,13 +3807,6 @@ void intel_power_domains_fini_hw(struct drm_i915_private *dev_priv) /* Remove the refcount we took to keep power well support disabled. */ if (!i915_modparams.disable_power_well) intel_display_power_put(dev_priv, POWER_DOMAIN_INIT); - - /* - * Remove the refcount we took in intel_runtime_pm_enable() in case - * the platform doesn't support runtime PM. - */ - if (!HAS_RUNTIME_PM(dev_priv)) - pm_runtime_put(kdev); } /** @@ -4074,3 +4065,18 @@ void intel_runtime_pm_enable(struct drm_i915_private *dev_priv) */ pm_runtime_put_autosuspend(kdev); } + +void intel_runtime_pm_disable(struct drm_i915_private *dev_priv) +{ + struct pci_dev *pdev = dev_priv->drm.pdev; + struct device *kdev = &pdev->dev; + + pm_runtime_dont_use_autosuspend(kdev); + + /* + * Remove the refcount we took in intel_runtime_pm_enable() in case + * the platform doesn't support runtime PM. + */ + if (!HAS_RUNTIME_PM(dev_priv)) + pm_runtime_put(kdev); +} From patchwork Thu Aug 9 09:16:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Wilson X-Patchwork-Id: 10561157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71F4614E2 for ; Thu, 9 Aug 2018 09:16:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6371329CFA for ; Thu, 9 Aug 2018 09:16:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 573292AB15; Thu, 9 Aug 2018 09:16:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C2C2729CFA for ; Thu, 9 Aug 2018 09:16:46 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1436D6E060; Thu, 9 Aug 2018 09:16:46 +0000 (UTC) X-Original-To: intel-gfx@lists.freedesktop.org Delivered-To: intel-gfx@lists.freedesktop.org Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192]) by gabe.freedesktop.org (Postfix) with ESMTPS id A55D76E060 for ; Thu, 9 Aug 2018 09:16:44 +0000 (UTC) X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Received: from haswell.alporthouse.com (unverified [78.156.65.138]) by fireflyinternet.com (Firefly Internet (M1)) with ESMTP id 12743771-1500050 for multiple; Thu, 09 Aug 2018 10:16:37 +0100 From: Chris Wilson To: intel-gfx@lists.freedesktop.org Date: Thu, 9 Aug 2018 10:16:35 +0100 Message-Id: <20180809091635.14976-2-chris@chris-wilson.co.uk> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180809091635.14976-1-chris@chris-wilson.co.uk> References: <20180809091635.14976-1-chris@chris-wilson.co.uk> Subject: [Intel-gfx] [PATCH 2/2] drm/i915: Track all held rpm wakerefs X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" X-Virus-Scanned: ClamAV using ClamSMTP Everytime we take a wakeref, record the stack trace of where it was taken; clearing the set if we ever drop back to no owners. For debugging a rpm leak, we can look at all the current wakerefs and check if they have a matching rpm_put. Signed-off-by: Chris Wilson --- drivers/gpu/drm/i915/Kconfig.debug | 13 +++ drivers/gpu/drm/i915/i915_drv.c | 3 +- drivers/gpu/drm/i915/i915_drv.h | 7 ++ drivers/gpu/drm/i915/intel_drv.h | 1 + drivers/gpu/drm/i915/intel_runtime_pm.c | 143 +++++++++++++++++++++++- 5 files changed, 165 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/i915/Kconfig.debug b/drivers/gpu/drm/i915/Kconfig.debug index 459f8f88a34c..ed572a454e01 100644 --- a/drivers/gpu/drm/i915/Kconfig.debug +++ b/drivers/gpu/drm/i915/Kconfig.debug @@ -39,6 +39,19 @@ config DRM_I915_DEBUG If in doubt, say "N". +config DRM_I915_DEBUG_RPM + bool "Insert extra checks into the runtime pm internals" + depends on DRM_I915 + default n + select STACKDEPOT + help + Enable extra sanity checks (including BUGs) along the runtime pm + paths that may slow the system down and if hit hang the machine. + + Recommended for driver developers only. + + If in doubt, say "N". + config DRM_I915_DEBUG_GEM bool "Insert extra checks into the GEM internals" default n diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c index 62ef105a241d..63992fe45dbf 100644 --- a/drivers/gpu/drm/i915/i915_drv.c +++ b/drivers/gpu/drm/i915/i915_drv.c @@ -1477,7 +1477,7 @@ void i915_driver_unload(struct drm_device *dev) intel_display_power_put(dev_priv, POWER_DOMAIN_INIT); - WARN_ON(atomic_read(&dev_priv->runtime_pm.wakeref_count)); + intel_runtime_pm_cleanup(dev_priv); } static void i915_driver_release(struct drm_device *dev) @@ -2603,6 +2603,7 @@ static int intel_runtime_suspend(struct device *kdev) DRM_DEBUG_KMS("Suspending device\n"); + intel_runtime_pm_cleanup(dev_priv); disable_rpm_wakeref_asserts(dev_priv); /* diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h index 0b10a30b7d96..c8d5b896b155 100644 --- a/drivers/gpu/drm/i915/i915_drv.h +++ b/drivers/gpu/drm/i915/i915_drv.h @@ -45,6 +45,7 @@ #include #include #include +#include #include #include @@ -1278,6 +1279,12 @@ struct i915_runtime_pm { atomic_t wakeref_count; bool suspended; bool irqs_enabled; + +#if IS_ENABLED(CONFIG_DRM_I915_DEBUG_RPM) + spinlock_t debug_lock; + depot_stack_handle_t *debug_owners; + unsigned long debug_count; +#endif }; enum intel_pipe_crc_source { diff --git a/drivers/gpu/drm/i915/intel_drv.h b/drivers/gpu/drm/i915/intel_drv.h index dc6c0cec9b36..968c9074f1a8 100644 --- a/drivers/gpu/drm/i915/intel_drv.h +++ b/drivers/gpu/drm/i915/intel_drv.h @@ -1957,6 +1957,7 @@ void bxt_display_core_init(struct drm_i915_private *dev_priv, bool resume); void bxt_display_core_uninit(struct drm_i915_private *dev_priv); void intel_runtime_pm_enable(struct drm_i915_private *dev_priv); void intel_runtime_pm_disable(struct drm_i915_private *dev_priv); +void intel_runtime_pm_cleanup(struct drm_i915_private *dev_priv); const char * intel_display_power_domain_str(enum intel_display_power_domain domain); diff --git a/drivers/gpu/drm/i915/intel_runtime_pm.c b/drivers/gpu/drm/i915/intel_runtime_pm.c index b78c3b48aa62..7f555a0ad2ee 100644 --- a/drivers/gpu/drm/i915/intel_runtime_pm.c +++ b/drivers/gpu/drm/i915/intel_runtime_pm.c @@ -49,6 +49,130 @@ * present for a given platform. */ +#if IS_ENABLED(CONFIG_DRM_I915_DEBUG_RPM) + +#include + +#define STACKDEPTH 12 + +static void track_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ + unsigned long entries[STACKDEPTH]; + struct stack_trace trace = { + .entries = entries, + .max_entries = ARRAY_SIZE(entries), + .skip = 2 + }; + unsigned long flags; + depot_stack_handle_t stack, *stacks; + + if (!HAS_RUNTIME_PM(i915)) + return; + + save_stack_trace(&trace); + if (trace.nr_entries && + trace.entries[trace.nr_entries - 1] == ULONG_MAX) + trace.nr_entries--; + + stack = depot_save_stack(&trace, GFP_NOWAIT | __GFP_NOWARN); + if (!stack) + return; + + spin_lock_irqsave(&i915->runtime_pm.debug_lock, flags); + stacks = krealloc(i915->runtime_pm.debug_owners, + (i915->runtime_pm.debug_count + 1) * sizeof(*stacks), + GFP_NOWAIT | __GFP_NOWARN); + if (stacks) { + stacks[i915->runtime_pm.debug_count++] = stack; + i915->runtime_pm.debug_owners = stacks; + } + spin_unlock_irqrestore(&i915->runtime_pm.debug_lock, flags); +} + +static void untrack_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ + depot_stack_handle_t *stacks; + unsigned long flags; + + spin_lock_irqsave(&i915->runtime_pm.debug_lock, flags); + stacks = fetch_and_zero(&i915->runtime_pm.debug_owners); + i915->runtime_pm.debug_count = 0; + spin_unlock_irqrestore(&i915->runtime_pm.debug_lock, flags); + + kfree(stacks); +} + +static int cmphandle(const void *_a, const void *_b) +{ + const depot_stack_handle_t * const a = _a, * const b = _b; + + if (*a < *b) + return -1; + else if (*a > *b) + return 1; + else + return 0; +} + +static void show_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ + unsigned long entries[STACKDEPTH]; + depot_stack_handle_t *stacks; + unsigned long flags, count, i; + char *buf; + + spin_lock_irqsave(&i915->runtime_pm.debug_lock, flags); + stacks = fetch_and_zero(&i915->runtime_pm.debug_owners); + count = fetch_and_zero(&i915->runtime_pm.debug_count); + spin_unlock_irqrestore(&i915->runtime_pm.debug_lock, flags); + if (!count) + return; + + DRM_DEBUG_DRIVER("leaked %lu wakerefs\n", count); + + buf = kmalloc(PAGE_SIZE, GFP_KERNEL); + if (!buf) + goto out_stacks; + + sort(stacks, count, sizeof(*stacks), cmphandle, NULL); + + for (i = 0; i < count; i++) { + struct stack_trace trace = { + .entries = entries, + .max_entries = ARRAY_SIZE(entries), + }; + depot_stack_handle_t stack = stacks[i]; + unsigned long rep; + + rep = 1; + while (i + 1 < count && stacks[i + 1] == stack) + rep++, i++; + depot_fetch_stack(stack, &trace); + snprint_stack_trace(buf, PAGE_SIZE, &trace, 0); + DRM_DEBUG_DRIVER("wakeref x%lu at\n%s", rep, buf); + } + + kfree(buf); +out_stacks: + kfree(stacks); +} + +#else + +static void track_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ +} + +static void untrack_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ +} + +static void show_intel_runtime_pm_wakeref(struct drm_i915_private *i915) +{ +} + +#endif + bool intel_display_power_well_is_enabled(struct drm_i915_private *dev_priv, enum i915_power_well_id power_well_id); @@ -3939,6 +4063,8 @@ void intel_runtime_pm_get(struct drm_i915_private *dev_priv) atomic_inc(&dev_priv->runtime_pm.wakeref_count); assert_rpm_wakelock_held(dev_priv); + + track_intel_runtime_pm_wakeref(dev_priv); } /** @@ -3973,6 +4099,8 @@ bool intel_runtime_pm_get_if_in_use(struct drm_i915_private *dev_priv) atomic_inc(&dev_priv->runtime_pm.wakeref_count); assert_rpm_wakelock_held(dev_priv); + track_intel_runtime_pm_wakeref(dev_priv); + return true; } @@ -4002,6 +4130,8 @@ void intel_runtime_pm_get_noresume(struct drm_i915_private *dev_priv) pm_runtime_get_noresume(kdev); atomic_inc(&dev_priv->runtime_pm.wakeref_count); + + track_intel_runtime_pm_wakeref(dev_priv); } /** @@ -4018,7 +4148,8 @@ void intel_runtime_pm_put(struct drm_i915_private *dev_priv) struct device *kdev = &pdev->dev; assert_rpm_wakelock_held(dev_priv); - atomic_dec(&dev_priv->runtime_pm.wakeref_count); + if (!atomic_dec_and_test(&dev_priv->runtime_pm.wakeref_count)) + untrack_intel_runtime_pm_wakeref(dev_priv); pm_runtime_mark_last_busy(kdev); pm_runtime_put_autosuspend(kdev); @@ -4080,3 +4211,13 @@ void intel_runtime_pm_disable(struct drm_i915_private *dev_priv) if (!HAS_RUNTIME_PM(dev_priv)) pm_runtime_put(kdev); } + +void intel_runtime_pm_cleanup(struct drm_i915_private *dev_priv) +{ + if (WARN_ON(atomic_read(&dev_priv->runtime_pm.wakeref_count))) { + show_intel_runtime_pm_wakeref(dev_priv); + atomic_set(&dev_priv->runtime_pm.wakeref_count, 0); + } + + untrack_intel_runtime_pm_wakeref(dev_priv); +}