From patchwork Fri Aug 3 08:26:45 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: jackm X-Patchwork-Id: 1269781 X-Patchwork-Delegate: roland@digitalvampire.org Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork2.kernel.org (Postfix) with ESMTP id 4A99CDF280 for ; Fri, 3 Aug 2012 08:27:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753007Ab2HCI1X (ORCPT ); Fri, 3 Aug 2012 04:27:23 -0400 Received: from eu1sys200aog119.obsmtp.com ([207.126.144.147]:60822 "HELO eu1sys200aog119.obsmtp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1752927Ab2HCI1N (ORCPT ); Fri, 3 Aug 2012 04:27:13 -0400 Received: from mtlsws123.lab.mtl.com ([82.166.227.17]) (using TLSv1) by eu1sys200aob119.postini.com ([207.126.147.11]) with SMTP ID DSNKUBuLWh6hpfhCGA5GJOVlpmE6JJJHb1iL@postini.com; Fri, 03 Aug 2012 08:27:10 UTC Received: from r-vnc04.lab.mtl.com (r-vnc04.lab.mtl.com [10.208.0.116]) by mtlsws123.lab.mtl.com (8.13.8/8.13.8) with ESMTP id q738R4Z0016220; Fri, 3 Aug 2012 11:27:04 +0300 From: Jack Morgenstein To: roland@kernel.org Cc: linux-rdma@vger.kernel.org, dotanb@mellanox.com, ogerlitz@mellanox.com, yevgenyp@mellanox.com, Jack Morgenstein Subject: [PATCH] IB/mlx4: fix possible deadlock with sm_lock spinlock Date: Fri, 3 Aug 2012 11:26:45 +0300 Message-Id: <1343982405-2828-1-git-send-email-jackm@dev.mellanox.co.il> X-Mailer: git-send-email 1.7.8.2 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The sm_lock spinlock is taken in the process context by mlx4_ib_modify_device, and in the interrupt context by update_sm_ah. Need to take that spinlock with irqsave, and release it with irqrestore. From a stack trace with LOCKDEP configured in the kernel: [ INFO: inconsistent lock state ] 3.5.0+ #20 Not tainted inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: (&(&ibdev->sm_lock)->rlock){?.+...}, at: [] update_sm_ah+0xad/0x100 [mlx4_ib] {HARDIRQ-ON-W} state was registered at: [] mark_irqflags+0x120/0x190 [] __lock_acquire+0x307/0x4c0 [] lock_acquire+0xb1/0x150 [] _raw_spin_lock+0x41/0x50 [] mlx4_ib_modify_device+0x63/0x240 [mlx4_ib] [] ib_modify_device+0x1c/0x20 [ib_core] [] set_node_desc+0x83/0xc0 [ib_core] [] dev_attr_store+0x20/0x30 [] sysfs_write_file+0xe6/0x170 [] vfs_write+0xc8/0x190 [] sys_write+0x51/0x90 [] system_call_fastpath+0x16/0x1b ... *** DEADLOCK *** 1 lock held by swapper/0/0: stack backtrace: Pid: 0, comm: swapper/0 Not tainted 3.5.0+ #20 Call Trace: [] print_usage_bug+0x18a/0x190 [] ? print_irq_inversion_bug+0x210/0x210 [] mark_lock_irq+0xf2/0x280 [] mark_lock+0x150/0x240 [] mark_irqflags+0x16f/0x190 [] __lock_acquire+0x307/0x4c0 [] ? update_sm_ah+0xad/0x100 [mlx4_ib] [] lock_acquire+0xb1/0x150 [] ? update_sm_ah+0xad/0x100 [mlx4_ib] [] _raw_spin_lock+0x41/0x50 [] ? update_sm_ah+0xad/0x100 [mlx4_ib] [] ? ib_create_ah+0x1a/0x40 [ib_core] [] update_sm_ah+0xad/0x100 [mlx4_ib] [] ? is_module_address+0x23/0x30 [] handle_port_mgmt_change_event+0xeb/0x150 [mlx4_ib] [] mlx4_ib_event+0x117/0x160 [mlx4_ib] [] ? _raw_spin_lock_irqsave+0x61/0x70 [] mlx4_dispatch_event+0x6c/0x90 [mlx4_core] [] mlx4_eq_int+0x500/0x950 [mlx4_core] Reported by: Or Gerlitz Signed-off-by: Jack Morgenstein Tested-by: Bart Van Assche --- drivers/infiniband/hw/mlx4/mad.c | 16 ++++++++++------ drivers/infiniband/hw/mlx4/main.c | 7 ++++--- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c index c27141f..9c2ae7e 100644 --- a/drivers/infiniband/hw/mlx4/mad.c +++ b/drivers/infiniband/hw/mlx4/mad.c @@ -125,6 +125,7 @@ static void update_sm_ah(struct mlx4_ib_dev *dev, u8 port_num, u16 lid, u8 sl) { struct ib_ah *new_ah; struct ib_ah_attr ah_attr; + unsigned long flags; if (!dev->send_agent[port_num - 1][0]) return; @@ -139,11 +140,11 @@ static void update_sm_ah(struct mlx4_ib_dev *dev, u8 port_num, u16 lid, u8 sl) if (IS_ERR(new_ah)) return; - spin_lock(&dev->sm_lock); + spin_lock_irqsave(&dev->sm_lock, flags); if (dev->sm_ah[port_num - 1]) ib_destroy_ah(dev->sm_ah[port_num - 1]); dev->sm_ah[port_num - 1] = new_ah; - spin_unlock(&dev->sm_lock); + spin_unlock_irqrestore(&dev->sm_lock, flags); } /* @@ -197,13 +198,15 @@ static void smp_snoop(struct ib_device *ibdev, u8 port_num, struct ib_mad *mad, static void node_desc_override(struct ib_device *dev, struct ib_mad *mad) { + unsigned long flags; + if ((mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_LID_ROUTED || mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) && mad->mad_hdr.method == IB_MGMT_METHOD_GET_RESP && mad->mad_hdr.attr_id == IB_SMP_ATTR_NODE_DESC) { - spin_lock(&to_mdev(dev)->sm_lock); + spin_lock_irqsave(&to_mdev(dev)->sm_lock, flags); memcpy(((struct ib_smp *) mad)->data, dev->node_desc, 64); - spin_unlock(&to_mdev(dev)->sm_lock); + spin_unlock_irqrestore(&to_mdev(dev)->sm_lock, flags); } } @@ -213,6 +216,7 @@ static void forward_trap(struct mlx4_ib_dev *dev, u8 port_num, struct ib_mad *ma struct ib_mad_send_buf *send_buf; struct ib_mad_agent *agent = dev->send_agent[port_num - 1][qpn]; int ret; + unsigned long flags; if (agent) { send_buf = ib_create_send_mad(agent, qpn, 0, 0, IB_MGMT_MAD_HDR, @@ -225,13 +229,13 @@ static void forward_trap(struct mlx4_ib_dev *dev, u8 port_num, struct ib_mad *ma * wrong following the IB spec strictly, but we know * it's OK for our devices). */ - spin_lock(&dev->sm_lock); + spin_lock_irqsave(&dev->sm_lock, flags); memcpy(send_buf->mad, mad, sizeof *mad); if ((send_buf->ah = dev->sm_ah[port_num - 1])) ret = ib_post_send_mad(send_buf, NULL); else ret = -EINVAL; - spin_unlock(&dev->sm_lock); + spin_unlock_irqrestore(&dev->sm_lock, flags); if (ret) ib_free_send_mad(send_buf); diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index fe2088c..7573304 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -423,16 +423,17 @@ static int mlx4_ib_modify_device(struct ib_device *ibdev, int mask, struct ib_device_modify *props) { struct mlx4_cmd_mailbox *mailbox; - + unsigned long flags; + if (mask & ~IB_DEVICE_MODIFY_NODE_DESC) return -EOPNOTSUPP; if (!(mask & IB_DEVICE_MODIFY_NODE_DESC)) return 0; - spin_lock(&to_mdev(ibdev)->sm_lock); + spin_lock_irqsave(&to_mdev(ibdev)->sm_lock, flags); memcpy(ibdev->node_desc, props->node_desc, 64); - spin_unlock(&to_mdev(ibdev)->sm_lock); + spin_unlock_irqrestore(&to_mdev(ibdev)->sm_lock, flags); /* * If possible, pass node desc to FW, so it can generate