btrfs: fix ABBA deadlock in btrfs_dev_replace_finishing()
diff mbox

Message ID 1411274509-10230-1-git-send-email-guaneryu@gmail.com
State New, archived
Headers show

Commit Message

Eryu Guan Sept. 21, 2014, 4:41 a.m. UTC
btrfs_map_bio() first calls btrfs_bio_counter_inc_blocked() which checks
fs state and increase bio_counter, then calls __btrfs_map_block() which
will take the dev_replace lock.

On the other hand, btrfs_dev_replace_finishing() takes dev_replace lock
first then set fs state to BTRFS_FS_STATE_DEV_REPLACING and waits for
bio_counter to be zero.

The deadlock can be reproduced easily by running replace and fsstress at
the same time, e.g.

mkfs -t btrfs -f /dev/sdb1 /dev/sdb2
mount /dev/sdb1 /mnt/btrfs
fsstress -d /mnt/btrfs -n 100 -p 2 -l 0 & # fsstress from ltp supports -l option
i=0
while btrfs replace start -Bf /dev/sdb2 /dev/sdb3 /mnt/btrfs && \
      btrfs replace start -Bf /dev/sdb3 /dev/sdb2 /mnt/btrfs; do
	echo "=== loop $i ==="
	let i=$i+1
done

This was introduced by

c404e0d Btrfs: fix use-after-free in the finishing procedure of the device replace

Signed-off-by: Eryu Guan <guaneryu@gmail.com>
---

Tested by the reproducer and xfstests, no new failure found.

But I found kmem_cache leak if I remove btrfs module after my new test case[1],
which does fsstress & replace & subvolume create/mount/umount/delete at the same
time.

BUG btrfs_extent_state (Tainted: G    B         ): Objects remaining in btrfs_extent_state on kmem_cache_close()
......
kmem_cache_destroy btrfs_extent_state: Slab cache still has objects
CPU: 3 PID: 9503 Comm: modprobe Tainted: G    B          3.17.0-rc5+ #12
Hardware name: Hewlett-Packard ProLiant DL388eGen8, BIOS P73 06/01/2012
 0000000000000000 000000008dd09c52 ffff880411c37eb0 ffffffff81642f7a
 ffff8800b9a19300 ffff880411c37ed0 ffffffff8118ce89 0000000000000000
 ffffffffa05dcd20 ffff880411c37ee0 ffffffffa056a80f ffff880411c37ef0
Call Trace:
 [<ffffffff81642f7a>] dump_stack+0x45/0x56
 [<ffffffff8118ce89>] kmem_cache_destroy+0xf9/0x100
 [<ffffffffa056a80f>] extent_io_exit+0x1f/0x50 [btrfs]
 [<ffffffffa05c3ae3>] exit_btrfs_fs+0x2c/0x549 [btrfs]
 [<ffffffff810efda2>] SyS_delete_module+0x162/0x200
 [<ffffffff81013bb7>] ? do_notify_resume+0x97/0xb0
 [<ffffffff8164af69>] system_call_fastpath+0x16/0x1b

The test would hang before the fix. I'm not sure if it's related to the fix
(seems not), please help review.

Thanks,
Eryu Guan

[1] http://www.spinics.net/lists/linux-btrfs/msg37625.html

 fs/btrfs/dev-replace.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

Comments

Miao Xie Sept. 21, 2014, 8:55 a.m. UTC | #1
It has been fixed by

https://patchwork.kernel.org/patch/4747961/

Thanks
Miao

On Sun, 21 Sep 2014 12:41:49 +0800, Eryu Guan wrote:
> btrfs_map_bio() first calls btrfs_bio_counter_inc_blocked() which checks
> fs state and increase bio_counter, then calls __btrfs_map_block() which
> will take the dev_replace lock.
> 
> On the other hand, btrfs_dev_replace_finishing() takes dev_replace lock
> first then set fs state to BTRFS_FS_STATE_DEV_REPLACING and waits for
> bio_counter to be zero.
> 
> The deadlock can be reproduced easily by running replace and fsstress at
> the same time, e.g.
> 
> mkfs -t btrfs -f /dev/sdb1 /dev/sdb2
> mount /dev/sdb1 /mnt/btrfs
> fsstress -d /mnt/btrfs -n 100 -p 2 -l 0 & # fsstress from ltp supports -l option
> i=0
> while btrfs replace start -Bf /dev/sdb2 /dev/sdb3 /mnt/btrfs && \
>       btrfs replace start -Bf /dev/sdb3 /dev/sdb2 /mnt/btrfs; do
> 	echo "=== loop $i ==="
> 	let i=$i+1
> done
> 
> This was introduced by
> 
> c404e0d Btrfs: fix use-after-free in the finishing procedure of the device replace
> 
> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
> ---
> 
> Tested by the reproducer and xfstests, no new failure found.
> 
> But I found kmem_cache leak if I remove btrfs module after my new test case[1],
> which does fsstress & replace & subvolume create/mount/umount/delete at the same
> time.
> 
> BUG btrfs_extent_state (Tainted: G    B         ): Objects remaining in btrfs_extent_state on kmem_cache_close()
> ......
> kmem_cache_destroy btrfs_extent_state: Slab cache still has objects
> CPU: 3 PID: 9503 Comm: modprobe Tainted: G    B          3.17.0-rc5+ #12
> Hardware name: Hewlett-Packard ProLiant DL388eGen8, BIOS P73 06/01/2012
>  0000000000000000 000000008dd09c52 ffff880411c37eb0 ffffffff81642f7a
>  ffff8800b9a19300 ffff880411c37ed0 ffffffff8118ce89 0000000000000000
>  ffffffffa05dcd20 ffff880411c37ee0 ffffffffa056a80f ffff880411c37ef0
> Call Trace:
>  [<ffffffff81642f7a>] dump_stack+0x45/0x56
>  [<ffffffff8118ce89>] kmem_cache_destroy+0xf9/0x100
>  [<ffffffffa056a80f>] extent_io_exit+0x1f/0x50 [btrfs]
>  [<ffffffffa05c3ae3>] exit_btrfs_fs+0x2c/0x549 [btrfs]
>  [<ffffffff810efda2>] SyS_delete_module+0x162/0x200
>  [<ffffffff81013bb7>] ? do_notify_resume+0x97/0xb0
>  [<ffffffff8164af69>] system_call_fastpath+0x16/0x1b
> 
> The test would hang before the fix. I'm not sure if it's related to the fix
> (seems not), please help review.
> 
> Thanks,
> Eryu Guan
> 
> [1] http://www.spinics.net/lists/linux-btrfs/msg37625.html
> 
>  fs/btrfs/dev-replace.c | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
> index eea26e1..5dfd292 100644
> --- a/fs/btrfs/dev-replace.c
> +++ b/fs/btrfs/dev-replace.c
> @@ -510,6 +510,7 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
>  	/* keep away write_all_supers() during the finishing procedure */
>  	mutex_lock(&root->fs_info->chunk_mutex);
>  	mutex_lock(&root->fs_info->fs_devices->device_list_mutex);
> +	btrfs_rm_dev_replace_blocked(fs_info);
>  	btrfs_dev_replace_lock(dev_replace);
>  	dev_replace->replace_state =
>  		scrub_ret ? BTRFS_IOCTL_DEV_REPLACE_STATE_CANCELED
> @@ -567,12 +568,8 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
>  	btrfs_kobj_rm_device(fs_info, src_device);
>  	btrfs_kobj_add_device(fs_info, tgt_device);
>  
> -	btrfs_rm_dev_replace_blocked(fs_info);
> -
>  	btrfs_rm_dev_replace_srcdev(fs_info, src_device);
>  
> -	btrfs_rm_dev_replace_unblocked(fs_info);
> -
>  	/*
>  	 * this is again a consistent state where no dev_replace procedure
>  	 * is running, the target device is part of the filesystem, the
> @@ -581,6 +578,7 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
>  	 * belong to this filesystem.
>  	 */
>  	btrfs_dev_replace_unlock(dev_replace);
> +	btrfs_rm_dev_replace_unblocked(fs_info);
>  	mutex_unlock(&root->fs_info->fs_devices->device_list_mutex);
>  	mutex_unlock(&root->fs_info->chunk_mutex);
>  
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch
diff mbox

diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
index eea26e1..5dfd292 100644
--- a/fs/btrfs/dev-replace.c
+++ b/fs/btrfs/dev-replace.c
@@ -510,6 +510,7 @@  static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 	/* keep away write_all_supers() during the finishing procedure */
 	mutex_lock(&root->fs_info->chunk_mutex);
 	mutex_lock(&root->fs_info->fs_devices->device_list_mutex);
+	btrfs_rm_dev_replace_blocked(fs_info);
 	btrfs_dev_replace_lock(dev_replace);
 	dev_replace->replace_state =
 		scrub_ret ? BTRFS_IOCTL_DEV_REPLACE_STATE_CANCELED
@@ -567,12 +568,8 @@  static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 	btrfs_kobj_rm_device(fs_info, src_device);
 	btrfs_kobj_add_device(fs_info, tgt_device);
 
-	btrfs_rm_dev_replace_blocked(fs_info);
-
 	btrfs_rm_dev_replace_srcdev(fs_info, src_device);
 
-	btrfs_rm_dev_replace_unblocked(fs_info);
-
 	/*
 	 * this is again a consistent state where no dev_replace procedure
 	 * is running, the target device is part of the filesystem, the
@@ -581,6 +578,7 @@  static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
 	 * belong to this filesystem.
 	 */
 	btrfs_dev_replace_unlock(dev_replace);
+	btrfs_rm_dev_replace_unblocked(fs_info);
 	mutex_unlock(&root->fs_info->fs_devices->device_list_mutex);
 	mutex_unlock(&root->fs_info->chunk_mutex);