From patchwork Thu Feb 10 07:36:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 12741516 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BA22C433EF for ; Thu, 10 Feb 2022 07:37:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236171AbiBJHhK (ORCPT ); Thu, 10 Feb 2022 02:37:10 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:48658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233771AbiBJHhJ (ORCPT ); Thu, 10 Feb 2022 02:37:09 -0500 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8533271 for ; Wed, 9 Feb 2022 23:37:10 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1644478627; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1XzKCGCgEDEV0dWUsbBGWzTEVpubfmjotpe20hxRjk0=; b=AY9myMgz4Bq3NhVoelfTFOYN4siUnR0N7ncUr8jxLaLMFDuAR10ErdWIkeGvLfFqpqw3SK odZ+Q3LGhzhQHS9EsQGnDx0za8IXQ2LXU73RxiILnJyUPRrMA+pJE/XftzEjm3Rkw5kxf1 pManFjZgwNZO9Y5YHaA7wavENvb/RcQ= From: Guoqing Jiang To: zyjzyj2000@gmail.com, jgg@ziepe.ca, rpearsonhpe@gmail.com Cc: linux-rdma@vger.kernel.org Subject: [PATCH 1/3] RDMA/rxe: Replace write_lock_bh with write_lock_irqsave in __rxe_add_index Date: Thu, 10 Feb 2022 15:36:53 +0800 Message-Id: <20220210073655.42281-2-guoqing.jiang@linux.dev> In-Reply-To: <20220210073655.42281-1-guoqing.jiang@linux.dev> References: <20220210073655.42281-1-guoqing.jiang@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org We need to make the lock fully IRQ safe, otherwise below calltrace appears. [ 495.697917] ------------[ cut here ]------------ [ 495.698316] WARNING: CPU: 5 PID: 67 at kernel/softirq.c:363 __local_bh_enable_ip+0xb1/0x110 [ ... ] [ 495.702594] CPU: 5 PID: 67 Comm: kworker/5:1 Kdump: loaded Tainted: G EL 5.17.0-rc3-57-default #17 [ 495.702856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 [ 495.703144] Workqueue: ib_cm cm_work_handler [ib_cm] [ 495.708238] RIP: 0010:__local_bh_enable_ip+0xb1/0x110 [ 495.713197] Code: e8 54 ae 04 00 e8 7f 4e 20 00 fb 66 0f 1f 44 00 00 65 8b 05 b1 03 f3 51 85 c0 74 51 5b 5d c3 65 8b 05 3f 0f f3 51 85 c0 75 8e <0f> 0b eb 8a e8 76 4c 20 00 eb 99 48 89 ef e8 9c 8d 0b 00 eb a2 48 [ 495.723257] RSP: 0018:ffff888100f9f1d8 EFLAGS: 00010046 [ 495.728296] RAX: 0000000000000000 RBX: 0000000000000201 RCX: dffffc0000000000 [ 495.733441] RDX: 0000000000000007 RSI: 0000000000000201 RDI: ffffffffb095dbac [ 495.738546] RBP: ffffffffc1761aa5 R08: ffffffffae1059da R09: 0000000000000000 [ 495.743689] R10: 0000000000000001 R11: 0000000000000001 R12: ffff88800f6cd380 [ 495.748913] R13: 0000000000000000 R14: ffff8880031e1ae0 R15: ffff8880031e1a28 [ 495.754091] FS: 0000000000000000(0000) GS:ffff888109880000(0000) knlGS:0000000000000000 [ 495.759217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 495.764434] CR2: 00007f69a232e830 CR3: 00000000b6a16005 CR4: 0000000000770ee0 [ 495.769531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 495.774505] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 495.779449] PKRU: 55555554 [ 495.784331] Call Trace: [ 495.789157] [ 495.793988] __rxe_add_index+0x35/0x40 [rdma_rxe] [ 495.798938] rxe_create_ah+0xa9/0x1e0 [rdma_rxe] [ 495.804007] _rdma_create_ah+0x28a/0x2c0 [ib_core] [ 495.809328] ? ib_create_srq_user+0x2c0/0x2c0 [ib_core] [ 495.814439] ? lock_acquire+0x182/0x410 [ 495.819558] ? lock_release+0x450/0x450 [ 495.824880] rdma_create_ah+0xe1/0x1a0 [ib_core] [ 495.830101] ? _rdma_create_ah+0x2c0/0x2c0 [ib_core] [ 495.835261] ? rwlock_bug.part.0+0x60/0x60 [ 495.840418] cm_alloc_msg+0xb4/0x260 [ib_cm] [ 495.845528] cm_alloc_priv_msg+0x29/0x70 [ib_cm] [ 495.850656] ib_send_cm_rep+0x7c/0x860 [ib_cm] [ 495.855677] ? lock_is_held_type+0xe4/0x140 [ 495.860761] rdma_accept+0x44c/0x5e0 [rdma_cm] [ 495.865817] ? cma_rep_recv+0x330/0x330 [rdma_cm] [ 495.870658] ? rcu_read_lock_sched_held+0x3f/0x60 [ 495.875388] ? trace_kmalloc+0x29/0xd0 [ 495.879807] ? __kmalloc+0x1c5/0x3a0 [ 495.884114] ? rtrs_iu_alloc+0x12b/0x260 [rtrs_core] [ 495.888343] rtrs_srv_rdma_cm_handler+0x7ba/0xcf0 [rtrs_server] [ 495.892503] ? rtrs_srv_inv_rkey_done+0x100/0x100 [rtrs_server] [ 495.896532] ? find_held_lock+0x85/0xa0 [ 495.900417] ? lock_release+0x24e/0x450 [ 495.904174] ? rdma_restrack_add+0x9c/0x220 [ib_core] [ 495.907939] ? rcu_read_lock_sched_held+0x3f/0x60 [ 495.911638] cma_cm_event_handler+0x77/0x2c0 [rdma_cm] [ 495.915225] cma_ib_req_handler+0xbd5/0x23f0 [rdma_cm] [ 495.918702] ? cma_cancel_operation+0x1f0/0x1f0 [rdma_cm] [ 495.922039] ? lockdep_lock+0xb4/0x170 [ 495.925195] ? _find_first_zero_bit+0x28/0x50 [ 495.928525] ? mark_held_locks+0x65/0x90 [ 495.931787] cm_process_work+0x2f/0x210 [ib_cm] [ 495.934952] ? _raw_spin_unlock_irq+0x35/0x50 [ 495.937930] ? cm_queue_work_unlock+0x40/0x110 [ib_cm] [ 495.940899] cm_req_handler+0xf7f/0x2030 [ib_cm] [ 495.943738] ? cm_lap_handler+0xba0/0xba0 [ib_cm] [ 495.946708] ? lockdep_hardirqs_on_prepare+0x220/0x220 [ 495.949600] cm_work_handler+0x6ce/0x37c0 [ib_cm] [ 495.952395] ? lock_acquire+0x182/0x410 [ 495.955245] ? lock_release+0x450/0x450 [ 495.958005] ? lock_downgrade+0x3c0/0x3c0 [ 495.960695] ? ib_cm_init_qp_attr+0xa90/0xa90 [ib_cm] [ 495.963323] ? mark_held_locks+0x24/0x90 [ 495.965902] ? lock_is_held_type+0xe4/0x140 [ 495.968597] process_one_work+0x5a8/0xa80 [ 495.971155] ? lock_release+0x450/0x450 [ 495.973812] ? pwq_dec_nr_in_flight+0x100/0x100 [ 495.976426] ? rwlock_bug.part.0+0x60/0x60 [ 495.979006] ? _raw_spin_lock_irq+0x54/0x60 [ 495.981600] worker_thread+0x2b5/0x760 [ 495.984272] ? process_one_work+0xa80/0xa80 [ 495.986832] kthread+0x169/0x1a0 [ 495.989348] ? kthread_complete_and_exit+0x20/0x20 [ 495.992032] ret_from_fork+0x1f/0x30 [ 495.994622] [ 495.997126] irq event stamp: 52525 [ 495.999637] hardirqs last enabled at (52523): [] _raw_spin_unlock_irqrestore+0x2d/0x60 [ 496.002367] hardirqs last disabled at (52524): [] _raw_spin_lock_irqsave+0x60/0x70 [ 496.005109] softirqs last enabled at (52514): [] rxe_post_recv+0xb8/0x120 [rdma_rxe] [ 496.007888] softirqs last disabled at (52525): [] __rxe_add_index+0x22/0x40 [rdma_rxe] [ 496.010698] ---[ end trace 0000000000000000 ]--- Signed-off-by: Guoqing Jiang Acked-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe_pool.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 63c594173565..b4444785da52 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -300,10 +300,11 @@ int __rxe_add_index(struct rxe_pool_elem *elem) { struct rxe_pool *pool = elem->pool; int err; + unsigned long flags; - write_lock_bh(&pool->pool_lock); + write_lock_irqsave(&pool->pool_lock, flags); err = __rxe_add_index_locked(elem); - write_unlock_bh(&pool->pool_lock); + write_unlock_irqrestore(&pool->pool_lock, flags); return err; } From patchwork Thu Feb 10 07:36:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 12741517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 486DAC433F5 for ; Thu, 10 Feb 2022 07:37:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236245AbiBJHhP (ORCPT ); Thu, 10 Feb 2022 02:37:15 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:48698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236181AbiBJHhM (ORCPT ); Thu, 10 Feb 2022 02:37:12 -0500 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9E043AD for ; Wed, 9 Feb 2022 23:37:13 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1644478630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r+a4LsXMhWI/zggLoZ4fcMr2yvNHQTd0ga7+aVIbSqI=; b=fLDeb7oL/+K/jb885/mxGA9dm0INYosTZEvYTgLsSGDgXF7X2Ngt4SJvm7EJ7c9dIm6ypE QHMPUh3LOKMqJqPI9MFEa+7JrvGdw7bN7ySZC6qSBBCq+4qhg8NCz4tq38U1RcQYoYpCMQ e5SQb2IKo4pDO7/t7t1HHImkQnjdOHA= From: Guoqing Jiang To: zyjzyj2000@gmail.com, jgg@ziepe.ca, rpearsonhpe@gmail.com Cc: linux-rdma@vger.kernel.org Subject: [PATCH 2/3] RDMA/rxe: Replace write_lock_bh with write_lock_irqsave in __rxe_drop_index Date: Thu, 10 Feb 2022 15:36:54 +0800 Message-Id: <20220210073655.42281-3-guoqing.jiang@linux.dev> In-Reply-To: <20220210073655.42281-1-guoqing.jiang@linux.dev> References: <20220210073655.42281-1-guoqing.jiang@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Same as __rxe_add_index, the lock need to be fully IRQ safe, otherwise below calltrace appears. [ 250.757218] ------------[ cut here ]------------ [ 250.758997] WARNING: CPU: 6 PID: 90 at kernel/softirq.c:363 __local_bh_enable_ip+0xb1/0x110 [ ... ] [ 250.769900] CPU: 6 PID: 90 Comm: kworker/u16:3 Kdump: loaded Tainted: G OEL 5.17.0-rc3-57-default #17 [ 250.770413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 [ 250.770955] Workqueue: ib_mad1 timeout_sends [ib_core] [ 250.771400] RIP: 0010:__local_bh_enable_ip+0xb1/0x110 [ 250.771678] Code: e8 54 ae 04 00 e8 7f 4e 20 00 fb 66 0f 1f 44 00 00 65 8b 05 b1 03 b3 60 85 c0 74 51 5b 5d c3 65 8b 05 3f 0f b3 60 85 c0 75 8e <0f> 0b eb 8a e8 76 4c 20 00 eb 99 48 89 ef e8 9c 8d 0b 00 eb a2 48 [ 250.772562] RSP: 0018:ffff88801b2e7ae8 EFLAGS: 00010046 [ 250.772845] RAX: 0000000000000000 RBX: 0000000000000201 RCX: dffffc0000000000 [ 250.773197] RDX: 0000000000000007 RSI: 0000000000000201 RDI: ffffffffa1d5dbac [ 250.773548] RBP: ffffffffc15c4da7 R08: ffffffff9f5059da R09: 0000000000000000 [ 250.773911] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 [ 250.774261] R13: ffff888017c0e850 R14: ffff888016f18000 R15: 0000000000000005 [ 250.774614] FS: 0000000000000000(0000) GS:ffff888104f00000(0000) knlGS:0000000000000000 [ 250.775009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 250.775296] CR2: 00007f7ea9e84fe8 CR3: 000000000216e002 CR4: 0000000000770ee0 [ 250.775651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 250.784298] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 250.791093] PKRU: 55555554 [ 250.796587] Call Trace: [ 250.801957] [ 250.807269] rxe_destroy_ah+0x17/0x60 [rdma_rxe] [ 250.812682] rdma_destroy_ah_user+0x5a/0xb0 [ib_core] [ 250.818242] cm_free_priv_msg+0x6e/0x130 [ib_cm] [ 250.823738] cm_send_handler+0x1f6/0x480 [ib_cm] [ 250.829176] ? ib_cm_insert_listen+0x100/0x100 [ib_cm] [ 250.834654] ? lockdep_hardirqs_on_prepare+0x129/0x220 [ 250.840111] ? _raw_spin_unlock_irqrestore+0x2d/0x60 [ 250.845514] timeout_sends+0x310/0x420 [ib_core] [ 250.851007] ? ib_send_mad+0x850/0x850 [ib_core] [ 250.856471] ? mark_held_locks+0x24/0x90 [ 250.861679] ? lock_is_held_type+0xe4/0x140 [ 250.866835] process_one_work+0x5a8/0xa80 [ 250.871949] ? lock_release+0x450/0x450 [ 250.877061] ? pwq_dec_nr_in_flight+0x100/0x100 [ 250.882144] ? rwlock_bug.part.0+0x60/0x60 [ 250.887093] ? _raw_spin_lock_irq+0x54/0x60 [ 250.891960] worker_thread+0x2b5/0x760 [ 250.896691] ? process_one_work+0xa80/0xa80 [ 250.901265] kthread+0x169/0x1a0 [ 250.905722] ? kthread_complete_and_exit+0x20/0x20 [ 250.910094] ret_from_fork+0x1f/0x30 [ 250.914381] [ 250.918441] irq event stamp: 21397 [ 250.922427] hardirqs last enabled at (21395): [] _raw_spin_unlock_irqrestore+0x2d/0x60 [ 250.926476] hardirqs last disabled at (21396): [] _raw_spin_lock_irq+0x54/0x60 [ 250.930443] softirqs last enabled at (21370): [] process_one_work+0x5a8/0xa80 [ 250.934329] softirqs last disabled at (21397): [] __rxe_drop_index+0x20/0x40 [rdma_rxe] [ 250.938120] ---[ end trace 0000000000000000 ]--- Signed-off-by: Guoqing Jiang Acked-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe_pool.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index b4444785da52..026b60363fd6 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -320,10 +320,11 @@ void __rxe_drop_index_locked(struct rxe_pool_elem *elem) void __rxe_drop_index(struct rxe_pool_elem *elem) { struct rxe_pool *pool = elem->pool; + unsigned long flags; - write_lock_bh(&pool->pool_lock); + write_lock_irqsave(&pool->pool_lock, flags); __rxe_drop_index_locked(elem); - write_unlock_bh(&pool->pool_lock); + write_unlock_irqrestore(&pool->pool_lock, flags); } void *rxe_alloc_locked(struct rxe_pool *pool) From patchwork Thu Feb 10 07:36:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 12741518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46B16C433F5 for ; Thu, 10 Feb 2022 07:37:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233771AbiBJHhS (ORCPT ); Thu, 10 Feb 2022 02:37:18 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:48744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236211AbiBJHhN (ORCPT ); Thu, 10 Feb 2022 02:37:13 -0500 Received: from out1.migadu.com (out1.migadu.com [IPv6:2001:41d0:2:863f::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2519BF4 for ; Wed, 9 Feb 2022 23:37:14 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1644478633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=evxpqzsNzQqSQfp0HWLeS9ofh9hXhS2Uc635wtLFMsU=; b=kQhZE9E4LUYUdBY2L4ETEDKzT2o28Q42X1UmDYhe3Ia1acahLMPFR7sAro4+Pg2/rXl/wg oobLrcoRoF9C0SV+a158/k2HOH6uRwnexclOC2yqk8glwNmJNmUP6ic5XakC7pIXcG8jik hlvwYD2SGD6AMgMs4und9OPCzPwHRAI= From: Guoqing Jiang To: zyjzyj2000@gmail.com, jgg@ziepe.ca, rpearsonhpe@gmail.com Cc: linux-rdma@vger.kernel.org Subject: [PATCH 3/3] RDMA/rxe: Replace spin_lock_bh with spin_lock_irqsave in post_one_send Date: Thu, 10 Feb 2022 15:36:55 +0800 Message-Id: <20220210073655.42281-4-guoqing.jiang@linux.dev> In-Reply-To: <20220210073655.42281-1-guoqing.jiang@linux.dev> References: <20220210073655.42281-1-guoqing.jiang@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Same as __rxe_add_index, the lock need to be fully IRQ safe, otherwise below calltrace appears. [ 763.942623] ------------[ cut here ]------------ [ 763.943171] WARNING: CPU: 5 PID: 97 at kernel/softirq.c:363 __local_bh_enable_ip+0xb1/0x110 [ ... ] [ 763.947276] CPU: 5 PID: 97 Comm: kworker/5:2 Kdump: loaded Tainted: G OEL 5.17.0-rc3-57-default #17 [ 763.947575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 [ 763.947893] Workqueue: ib_cm cm_work_handler [ib_cm] [ 763.948075] RIP: 0010:__local_bh_enable_ip+0xb1/0x110 [ 763.948232] Code: e8 54 ae 04 00 e8 7f 4e 20 00 fb 66 0f 1f 44 00 00 65 8b 05 b1 03 f3 56 85 c0 74 51 5b 5d c3 65 8b 05 3f 0f f3 56 85 c0 75 8e <0f> 0b eb 8a e8 76 4c 20 00 eb 99 48 89 ef e8 9c 8d 0b 00 eb a2 48 [ 763.948736] RSP: 0018:ffff888004a970e8 EFLAGS: 00010046 [ 763.948897] RAX: 0000000000000000 RBX: 0000000000000201 RCX: dffffc0000000000 [ 763.949095] RDX: 0000000000000007 RSI: 0000000000000201 RDI: ffffffffab95dbac [ 763.949292] RBP: ffffffffc127c269 R08: ffffffffa91059da R09: ffff88800afde323 [ 763.949556] R10: ffffed10015fbc64 R11: 0000000000000001 R12: ffffc900005a2000 [ 763.949781] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88800afde000 [ 763.949982] FS: 0000000000000000(0000) GS:ffff888104c80000(0000) knlGS:0000000000000000 [ 763.950205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 763.950367] CR2: 00007f85ec3f5b18 CR3: 0000000116216005 CR4: 0000000000770ee0 [ 763.956480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 763.962608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 763.968785] PKRU: 55555554 [ 763.974707] Call Trace: [ 763.980557] [ 763.986377] rxe_post_send+0x569/0x8e0 [rdma_rxe] [ 763.992340] ib_send_mad+0x4c1/0x850 [ib_core] [ 763.998442] ? ib_register_mad_agent+0x1710/0x1710 [ib_core] [ 764.004486] ? __kmalloc+0x21d/0x3a0 [ 764.010465] ib_post_send_mad+0x28c/0x10b0 [ib_core] [ 764.016480] ? lock_is_held_type+0xe4/0x140 [ 764.022359] ? find_held_lock+0x85/0xa0 [ 764.028230] ? lock_release+0x24e/0x450 [ 764.034061] ? timeout_sends+0x420/0x420 [ib_core] [ 764.039879] ? ib_create_send_mad+0x541/0x670 [ib_core] [ 764.045604] ? do_raw_spin_unlock+0x86/0xf0 [ 764.051178] ? preempt_count_sub+0x14/0xc0 [ 764.056851] ? lock_is_held_type+0xe4/0x140 [ 764.062412] ib_send_cm_rep+0x47a/0x860 [ib_cm] [ 764.067965] rdma_accept+0x44c/0x5e0 [rdma_cm] [ 764.073381] ? cma_rep_recv+0x330/0x330 [rdma_cm] [ 764.078762] ? rcu_read_lock_sched_held+0x3f/0x60 [ 764.084072] ? trace_kmalloc+0x29/0xd0 [ 764.089185] ? __kmalloc+0x1c5/0x3a0 [ 764.094185] ? rtrs_iu_alloc+0x12b/0x260 [rtrs_core] [ 764.099075] rtrs_srv_rdma_cm_handler+0x7ba/0xcf0 [rtrs_server] [ 764.103917] ? rtrs_srv_inv_rkey_done+0x100/0x100 [rtrs_server] [ 764.108563] ? find_held_lock+0x85/0xa0 [ 764.113033] ? lock_release+0x24e/0x450 [ 764.117452] ? rdma_restrack_add+0x9c/0x220 [ib_core] [ 764.121797] ? rcu_read_lock_sched_held+0x3f/0x60 [ 764.125961] cma_cm_event_handler+0x77/0x2c0 [rdma_cm] [ 764.130061] cma_ib_req_handler+0xbd5/0x23f0 [rdma_cm] [ 764.134027] ? cma_cancel_operation+0x1f0/0x1f0 [rdma_cm] [ 764.137950] ? lockdep_lock+0xb4/0x170 [ 764.141667] ? _find_first_zero_bit+0x28/0x50 [ 764.145486] ? mark_held_locks+0x65/0x90 [ 764.149002] cm_process_work+0x2f/0x210 [ib_cm] [ 764.152413] ? _raw_spin_unlock_irq+0x35/0x50 [ 764.155763] ? cm_queue_work_unlock+0x40/0x110 [ib_cm] [ 764.159080] cm_req_handler+0xf7f/0x2030 [ib_cm] [ 764.162522] ? cm_lap_handler+0xba0/0xba0 [ib_cm] [ 764.165847] ? lockdep_hardirqs_on_prepare+0x220/0x220 [ 764.169155] cm_work_handler+0x6ce/0x37c0 [ib_cm] [ 764.172497] ? lock_acquire+0x182/0x410 [ 764.175771] ? lock_release+0x450/0x450 [ 764.178925] ? lock_downgrade+0x3c0/0x3c0 [ 764.182148] ? ib_cm_init_qp_attr+0xa90/0xa90 [ib_cm] [ 764.185511] ? mark_held_locks+0x24/0x90 [ 764.188692] ? lock_is_held_type+0xe4/0x140 [ 764.191876] process_one_work+0x5a8/0xa80 [ 764.195034] ? lock_release+0x450/0x450 [ 764.198208] ? pwq_dec_nr_in_flight+0x100/0x100 [ 764.201433] ? rwlock_bug.part.0+0x60/0x60 [ 764.204660] ? _raw_spin_lock_irq+0x54/0x60 [ 764.207835] worker_thread+0x2b5/0x760 [ 764.210920] ? process_one_work+0xa80/0xa80 [ 764.214014] kthread+0x169/0x1a0 [ 764.217033] ? kthread_complete_and_exit+0x20/0x20 [ 764.220205] ret_from_fork+0x1f/0x30 [ 764.223467] [ 764.226482] irq event stamp: 55805 [ 764.229527] hardirqs last enabled at (55803): [] _raw_spin_unlock_irqrestore+0x2d/0x60 [ 764.232779] hardirqs last disabled at (55804): [] _raw_spin_lock_irqsave+0x60/0x70 [ 764.236052] softirqs last enabled at (55794): [] rxe_post_recv+0xb8/0x120 [rdma_rxe] [ 764.239428] softirqs last disabled at (55805): [] rxe_post_send+0x1eb/0x8e0 [rdma_rxe] [ 764.242740] ---[ end trace 0000000000000000 ]--- Signed-off-by: Guoqing Jiang Acked-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe_verbs.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 9f0aef4b649d..0056418425a1 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -644,12 +644,13 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr, struct rxe_sq *sq = &qp->sq; struct rxe_send_wqe *send_wqe; int full; + unsigned long flags; err = validate_send_wr(qp, ibwr, mask, length); if (err) return err; - spin_lock_bh(&qp->sq.sq_lock); + spin_lock_irqsave(&qp->sq.sq_lock, flags); full = queue_full(sq->queue, QUEUE_TYPE_TO_DRIVER); @@ -663,7 +664,7 @@ static int post_one_send(struct rxe_qp *qp, const struct ib_send_wr *ibwr, queue_advance_producer(sq->queue, QUEUE_TYPE_TO_DRIVER); - spin_unlock_bh(&qp->sq.sq_lock); + spin_unlock_irqrestore(&qp->sq.sq_lock, flags); return 0; }