From patchwork Tue Aug 27 20:12:59 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Hurley X-Patchwork-Id: 2850308 Return-Path: X-Original-To: patchwork-dri-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 1CBEBBF546 for ; Tue, 27 Aug 2013 20:24:01 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A9AE1204B2 for ; Tue, 27 Aug 2013 20:23:59 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) by mail.kernel.org (Postfix) with ESMTP id 0F8F420498 for ; Tue, 27 Aug 2013 20:23:58 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 58631E5C3E for ; Tue, 27 Aug 2013 13:23:57 -0700 (PDT) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from mailout02.c08.mtsvc.net (mailout02.c08.mtsvc.net [205.186.168.190]) by gabe.freedesktop.org (Postfix) with ESMTP id BA092E769E for ; Tue, 27 Aug 2013 13:13:58 -0700 (PDT) Received: from n18.c08.mtsvc.net ([205.186.176.18]) by mailout02.c08.mtsvc.net with esmtp (Exim 4.72) (envelope-from ) id 1VEPeD-0001uo-2T; Tue, 27 Aug 2013 13:13:54 -0700 Received: from 68-184-16-174.dhcp.unas.ma.charter.com ([68.184.16.174]:56256 helo=thor.lan) by n18.c08.mtsvc.net with esmtpsa (TLS1.1:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.80.1) (envelope-from ) id 1VEPe9-0001XP-SJ; Tue, 27 Aug 2013 13:13:52 -0700 From: Peter Hurley To: Dave Airlie , Ben Skeggs Subject: [PATCH 6/9] drm/nouveau: Convert event handler list to RCU Date: Tue, 27 Aug 2013 16:12:59 -0400 Message-Id: <1377634382-13872-7-git-send-email-peter@hurleysoftware.com> X-Mailer: git-send-email 1.8.1.2 In-Reply-To: <1377634382-13872-1-git-send-email-peter@hurleysoftware.com> References: <1377634382-13872-1-git-send-email-peter@hurleysoftware.com> X-Authenticated-User: 125194 peter@hurleysoftware.com X-Spam-Status: No, score=-6.7 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Level: Cc: nouveau@lists.freedesktop.org, Peter Hurley , dri-devel@lists.freedesktop.org X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dri-devel-bounces+patchwork-dri-devel=patchwork.kernel.org@lists.freedesktop.org Errors-To: dri-devel-bounces+patchwork-dri-devel=patchwork.kernel.org@lists.freedesktop.org X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Lockdep report [1] correctly identifies a potential deadlock between nouveau_drm_vblank_enable() and nouveau_drm_vblank_handler() due to inverted lock order of event->lock with drm core's dev->vblank_time_lock. Fix vblank event deadlock by converting event handler list to RCU. List updates remain serialized by event->lock. List traversal & handler execution is lockless which prevents inverted lock order problems with the drm core. Safe deletion of the event handler is accomplished with synchronize_rcu() for those handlers stored in nouveau_object-based containers (nouveau_drm & nv50_/nvc0_software_context); otherwise, with kfree_rcu (for nouveau_connector & uevent temporary handlers). [1] ====================================================== [ INFO: possible circular locking dependency detected ] 3.10.0-0+tip-xeon+lockdep #0+tip Not tainted ------------------------------------------------------- swapper/7/0 is trying to acquire lock: (&(&dev->vblank_time_lock)->rlock){-.....}, at: [] drm_handle_vblank+0x69/0x400 [drm] but task is already holding lock: (&(&event->lock)->rlock){-.....}, at: [] nouveau_event_trigger+0x4d/0xd0 [nouveau] which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&(&event->lock)->rlock){-.....}: [] lock_acquire+0x92/0x1f0 [] _raw_spin_lock_irqsave+0x46/0x60 [] nouveau_event_get+0x27/0xa0 [nouveau] [] nouveau_drm_vblank_enable+0x8d/0xf0 [nouveau] [] drm_vblank_get+0xf8/0x2c0 [drm] [] drm_wait_vblank+0x84/0x6f0 [drm] [] drm_ioctl+0x559/0x690 [drm] [] do_vfs_ioctl+0x97/0x590 [] SyS_ioctl+0x91/0xb0 [] system_call_fastpath+0x16/0x1b -> #0 (&(&dev->vblank_time_lock)->rlock){-.....}: [] __lock_acquire+0x1c43/0x1d30 [] lock_acquire+0x92/0x1f0 [] _raw_spin_lock_irqsave+0x46/0x60 [] drm_handle_vblank+0x69/0x400 [drm] [] nouveau_drm_vblank_handler+0x27/0x30 [nouveau] [] nouveau_event_trigger+0x90/0xd0 [nouveau] [] nv50_disp_intr+0xdd/0x230 [nouveau] [] nouveau_mc_intr+0xa1/0x100 [nouveau] [] handle_irq_event_percpu+0x6c/0x3d0 [] handle_irq_event+0x48/0x70 [] handle_fasteoi_irq+0x5a/0x100 [] handle_irq+0x22/0x40 [] do_IRQ+0x5a/0xd0 [] ret_from_intr+0x0/0x13 [] arch_cpu_idle+0x26/0x30 [] cpu_startup_entry+0xce/0x410 [] start_secondary+0x255/0x25c other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&(&event->lock)->rlock); lock(&(&dev->vblank_time_lock)->rlock); lock(&(&event->lock)->rlock); lock(&(&dev->vblank_time_lock)->rlock); *** DEADLOCK *** 1 lock held by swapper/7/0: #0: (&(&event->lock)->rlock){-.....}, at: [] nouveau_event_trigger+0x4d/0xd0 [nouveau] stack backtrace: CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0-0+tip-xeon+lockdep #0+tip Hardware name: Dell Inc. Precision WorkStation T5400 /0RW203, BIOS A11 04/30/2012 ffffffff821ccf60 ffff8802bfdc3b08 ffffffff81755f39 ffff8802bfdc3b58 ffffffff8174f526 0000000000000002 ffff8802bfdc3be8 ffff8802b330a7f8 ffff8802b330a820 ffff8802b330a7f8 0000000000000000 0000000000000001 Call Trace: [] dump_stack+0x19/0x1b [] print_circular_bug+0x1fb/0x20c [] __lock_acquire+0x1c43/0x1d30 [] ? trace_hardirqs_off+0xd/0x10 [] ? flat_send_IPI_mask+0x88/0xa0 [] lock_acquire+0x92/0x1f0 [] ? drm_handle_vblank+0x69/0x400 [drm] [] _raw_spin_lock_irqsave+0x46/0x60 [] ? drm_handle_vblank+0x69/0x400 [drm] [] drm_handle_vblank+0x69/0x400 [drm] [] nouveau_drm_vblank_handler+0x27/0x30 [nouveau] [] nouveau_event_trigger+0x90/0xd0 [nouveau] [] nv50_disp_intr+0xdd/0x230 [nouveau] [] ? _raw_spin_unlock_irqrestore+0x42/0x80 [] ? __hrtimer_start_range_ns+0x16c/0x530 [] nouveau_mc_intr+0xa1/0x100 [nouveau] [] handle_irq_event_percpu+0x6c/0x3d0 [] handle_irq_event+0x48/0x70 [] ? handle_fasteoi_irq+0x1e/0x100 [] handle_fasteoi_irq+0x5a/0x100 [] handle_irq+0x22/0x40 [] do_IRQ+0x5a/0xd0 [] common_interrupt+0x6f/0x6f [] ? default_idle+0x1d/0x290 [] ? default_idle+0x1b/0x290 [] arch_cpu_idle+0x26/0x30 [] cpu_startup_entry+0xce/0x410 [] ? clockevents_register_device+0xdc/0x140 [] start_secondary+0x255/0x25c Reported-by: Dave Jones Signed-off-by: Peter Hurley --- drivers/gpu/drm/nouveau/core/core/event.c | 22 +++++++++++----------- .../gpu/drm/nouveau/core/engine/software/nv50.c | 1 + .../gpu/drm/nouveau/core/engine/software/nvc0.c | 1 + drivers/gpu/drm/nouveau/core/include/core/event.h | 1 + drivers/gpu/drm/nouveau/nouveau_connector.c | 2 +- drivers/gpu/drm/nouveau/nouveau_drm.c | 1 + 6 files changed, 16 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/nouveau/core/core/event.c b/drivers/gpu/drm/nouveau/core/core/event.c index 4cd1ebe..ce0a0ef 100644 --- a/drivers/gpu/drm/nouveau/core/core/event.c +++ b/drivers/gpu/drm/nouveau/core/core/event.c @@ -22,6 +22,7 @@ #include #include +#include void nouveau_event_handler_install(struct nouveau_event *event, int index, @@ -37,7 +38,7 @@ nouveau_event_handler_install(struct nouveau_event *event, int index, handler->priv = priv; spin_lock_irqsave(&event->lock, flags); - list_add(&handler->head, &event->index[index].list); + list_add_rcu(&handler->head, &event->index[index].list); spin_unlock_irqrestore(&event->lock, flags); } @@ -51,7 +52,7 @@ nouveau_event_handler_remove(struct nouveau_event *event, int index, return; spin_lock_irqsave(&event->lock, flags); - list_del(&handler->head); + list_del_rcu(&handler->head); spin_unlock_irqrestore(&event->lock, flags); } @@ -71,7 +72,7 @@ nouveau_event_handler_create(struct nouveau_event *event, int index, __set_bit(NVKM_EVENT_ENABLE, &handler->flags); spin_lock_irqsave(&event->lock, flags); - list_add(&handler->head, &event->index[index].list); + list_add_rcu(&handler->head, &event->index[index].list); if (!event->index[index].refs++) { if (event->enable) event->enable(event, index); @@ -94,9 +95,9 @@ nouveau_event_handler_destroy(struct nouveau_event *event, int index, if (event->disable) event->disable(event, index); } - list_del(&handler->head); + list_del_rcu(&handler->head); spin_unlock_irqrestore(&event->lock, flags); - kfree(handler); + kfree_rcu(handler, rcu); } static void @@ -147,20 +148,19 @@ nouveau_event_get(struct nouveau_event *event, int index, void nouveau_event_trigger(struct nouveau_event *event, int index) { - struct nouveau_eventh *handler, *temp; - unsigned long flags; + struct nouveau_eventh *handler; if (index >= event->index_nr) return; - spin_lock_irqsave(&event->lock, flags); - list_for_each_entry_safe(handler, temp, &event->index[index].list, head) { + rcu_read_lock(); + list_for_each_entry_rcu(handler, &event->index[index].list, head) { if (test_bit(NVKM_EVENT_ENABLE, &handler->flags)) { if (handler->func(handler, index) == NVKM_EVENT_DROP) - nouveau_event_put_locked(event, index, handler); + nouveau_event_put(event, index, handler); } } - spin_unlock_irqrestore(&event->lock, flags); + rcu_read_unlock(); } void diff --git a/drivers/gpu/drm/nouveau/core/engine/software/nv50.c b/drivers/gpu/drm/nouveau/core/engine/software/nv50.c index dfce1f5..87aeee1 100644 --- a/drivers/gpu/drm/nouveau/core/engine/software/nv50.c +++ b/drivers/gpu/drm/nouveau/core/engine/software/nv50.c @@ -191,6 +191,7 @@ nv50_software_context_dtor(struct nouveau_object *object) nouveau_event_handler_remove(disp->vblank, i, &chan->base.vblank.event[i]); } + synchronize_rcu(); _nouveau_software_context_dtor(object); } diff --git a/drivers/gpu/drm/nouveau/core/engine/software/nvc0.c b/drivers/gpu/drm/nouveau/core/engine/software/nvc0.c index d1f52c0..e87ba42 100644 --- a/drivers/gpu/drm/nouveau/core/engine/software/nvc0.c +++ b/drivers/gpu/drm/nouveau/core/engine/software/nvc0.c @@ -197,6 +197,7 @@ nvc0_software_context_dtor(struct nouveau_object *object) nouveau_event_handler_remove(disp->vblank, i, &chan->base.vblank.event[i]); } + synchronize_rcu(); _nouveau_software_context_dtor(object); } diff --git a/drivers/gpu/drm/nouveau/core/include/core/event.h b/drivers/gpu/drm/nouveau/core/include/core/event.h index 45e8a0f..f01b173 100644 --- a/drivers/gpu/drm/nouveau/core/include/core/event.h +++ b/drivers/gpu/drm/nouveau/core/include/core/event.h @@ -13,6 +13,7 @@ struct nouveau_eventh { unsigned long flags; void *priv; int (*func)(struct nouveau_eventh *, int index); + struct rcu_head rcu; }; struct nouveau_event { diff --git a/drivers/gpu/drm/nouveau/nouveau_connector.c b/drivers/gpu/drm/nouveau/nouveau_connector.c index 13b38ed..14fce8a 100644 --- a/drivers/gpu/drm/nouveau/nouveau_connector.c +++ b/drivers/gpu/drm/nouveau/nouveau_connector.c @@ -106,7 +106,7 @@ nouveau_connector_destroy(struct drm_connector *connector) kfree(nv_connector->edid); drm_sysfs_connector_remove(connector); drm_connector_cleanup(connector); - kfree(connector); + kfree_rcu(nv_connector, hpd_func.rcu); } static struct nouveau_i2c_port * diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c index 4187cad..544ca19 100644 --- a/drivers/gpu/drm/nouveau/nouveau_drm.c +++ b/drivers/gpu/drm/nouveau/nouveau_drm.c @@ -432,6 +432,7 @@ nouveau_drm_unload(struct drm_device *dev) for (i = 0; i < ARRAY_SIZE(drm->vblank); i++) nouveau_event_handler_remove(disp->vblank, i, &drm->vblank[i]); + synchronize_rcu(); nouveau_cli_destroy(&drm->client); return 0;