From patchwork Thu Oct 1 08:31:51 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junichi Nomura X-Patchwork-Id: 7306361 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Original-To: patchwork-dm-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 890459F314 for ; Thu, 1 Oct 2015 08:37:27 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 98A4F207B5 for ; Thu, 1 Oct 2015 08:37:26 +0000 (UTC) Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6DD73207B1 for ; Thu, 1 Oct 2015 08:37:25 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t918XkZr043257; Thu, 1 Oct 2015 04:33:47 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id t918Xi2b027243 for ; Thu, 1 Oct 2015 04:33:44 -0400 Received: from mx1.redhat.com (ext-mx05.extmail.prod.ext.phx2.redhat.com [10.5.110.29]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t918XiGq032219 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 1 Oct 2015 04:33:44 -0400 Received: from tyo202.gate.nec.co.jp (TYO202.gate.nec.co.jp [210.143.35.52]) by mx1.redhat.com (Postfix) with ESMTPS id 8C037461E2; Thu, 1 Oct 2015 08:33:39 +0000 (UTC) Received: from mailgate3.nec.co.jp ([10.7.69.192]) by tyo202.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id t918XbbE016793; Thu, 1 Oct 2015 17:33:37 +0900 (JST) Received: from mailsv4.nec.co.jp (imss61.nec.co.jp [10.7.69.156]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id t918Xb209904; Thu, 1 Oct 2015 17:33:37 +0900 (JST) Received: from mail02.kamome.nec.co.jp (mail02.kamome.nec.co.jp [10.25.43.5]) by mailsv4.nec.co.jp (8.13.8/8.13.4) with ESMTP id t918XaTe026759; Thu, 1 Oct 2015 17:33:37 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.140] [10.38.151.140]) by mail01b.kamome.nec.co.jp with ESMTP id BT-MMP-2333452; Thu, 1 Oct 2015 17:31:52 +0900 Received: from BPXM12GP.gisp.nec.co.jp ([169.254.2.181]) by BPXC12GP.gisp.nec.co.jp ([10.38.151.140]) with mapi id 14.03.0224.002; Thu, 1 Oct 2015 17:31:51 +0900 From: Junichi Nomura To: device-mapper development , Mikulas Patocka Thread-Topic: [PATCH] dm: fix AB-BA deadlock in __dm_destroy() Thread-Index: AQHQ/COehE+RYpLz4kSlLXxs96viRw== Date: Thu, 1 Oct 2015 08:31:51 +0000 Message-ID: <20151001083149.GA13075@xzibit.linux.bs1.fc.nec.co.jp> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.34.125.85] Content-ID: <90D2418C22844A4F947D211F0ABC9344@gisp.nec.co.jp> MIME-Version: 1.0 X-RedHat-Spam-Score: -1.901 (BAYES_50, DCC_REPUT_00_12, RCVD_IN_DNSWL_MED, SPF_HELO_PASS, SPF_PASS, URIBL_BLOCKED) 210.143.35.52 TYO202.gate.nec.co.jp 210.143.35.52 TYO202.gate.nec.co.jp X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Scanned-By: MIMEDefang 2.78 on 10.5.110.29 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id t918Xi2b027243 X-loop: dm-devel@redhat.com Subject: [dm-devel] [PATCH] dm: fix AB-BA deadlock in __dm_destroy() X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP __dm_destroy() takes io_barrier SRCU lock (dm_get_live_table) and suspend_lock in reverse order. That can cause AB-BA deadlock: Example: __dm_destroy dm_swap_table --------------------------------------------------- mutex_lock(suspend_lock) dm_get_live_table() srcu_read_lock(io_barrier) dm_sync_table() synchronize_srcu(io_barrier) .. waiting for dm_put_live_table() mutex_lock(suspend_lock) .. waiting for suspend_lock This patch fixes the lock ordering. Signed-off-by: Jun'ichi Nomura Fixes: ab7c7bb6f4ab ("dm: hold suspend_lock while suspending device during device deletion") Cc: Mikulas Patocka Acked-by: Mikulas Patocka --- The problem could be reproduced with this script but it might take long. (In my environment, it took more than 10 minutes) -- cut here -- #!/bin/bash t0="0 1024 zero" t1="0 1024 error" mapname=testmap work1() { while true; do dmsetup create --notable $mapname echo "$t0" | dmsetup load $mapname dmsetup resume $mapname dmsetup remove_all done } work2() { while true; do echo "$t1" | dmsetup load $mapname dmsetup resume $mapname echo "$t0" | dmsetup load $mapname dmsetup resume $mapname done } work1 & work2 & wait -- cut here -- When starting the script, it will emit a lot of errors such as "No such device or address" and stops when the deadlock occurs. Backtrace of dmsetup will look like this: # ps auxw|grep dmsetup root 32209 0.0 0.0 130024 3060 pts/0 D+ 03:26 0:00 dmsetup resume testmap root 32210 0.0 0.0 130024 3048 pts/0 D+ 03:26 0:00 dmsetup remove_all # cat /proc/32210/stack [] __dm_destroy+0xba/0x280 [dm_mod] [] dm_destroy+0x13/0x20 [dm_mod] [] dm_hash_remove_all+0x6d/0x130 [dm_mod] [] remove_all+0x22/0x30 [dm_mod] [] ctl_ioctl+0x255/0x4d0 [dm_mod] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [] do_vfs_ioctl+0x2d2/0x4b0 [] SyS_ioctl+0x79/0x90 [] entry_SYSCALL_64_fastpath+0x12/0x71 [] 0xffffffffffffffff # cat /proc/32209/stack [] __synchronize_srcu+0xf4/0x130 [] synchronize_srcu+0x24/0x30 [] dm_swap_table+0x17d/0x2e0 [dm_mod] [] dev_suspend+0x9a/0x240 [dm_mod] [] ctl_ioctl+0x255/0x4d0 [dm_mod] [] dm_ctl_ioctl+0x13/0x20 [dm_mod] [] do_vfs_ioctl+0x2d2/0x4b0 [] SyS_ioctl+0x79/0x90 [] entry_SYSCALL_64_fastpath+0x12/0x71 [] 0xffffffffffffffff -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 6264781..7289ece 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2837,8 +2837,6 @@ static void __dm_destroy(struct mapped_device *md, bool wait) might_sleep(); - map = dm_get_live_table(md, &srcu_idx); - spin_lock(&_minor_lock); idr_replace(&_minor_idr, MINOR_ALLOCED, MINOR(disk_devt(dm_disk(md)))); set_bit(DMF_FREEING, &md->flags); @@ -2852,14 +2850,14 @@ static void __dm_destroy(struct mapped_device *md, bool wait) * do not race with internal suspend. */ mutex_lock(&md->suspend_lock); + map = dm_get_live_table(md, &srcu_idx); if (!dm_suspended_md(md)) { dm_table_presuspend_targets(map); dm_table_postsuspend_targets(map); } - mutex_unlock(&md->suspend_lock); - /* dm_put_live_table must be before msleep, otherwise deadlock is possible */ dm_put_live_table(md, srcu_idx); + mutex_unlock(&md->suspend_lock); /* * Rare, but there may be I/O requests still going to complete,