From patchwork Mon Apr 18 08:51:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anand Jain X-Patchwork-Id: 8868771 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id DDBFBBF29F for ; Mon, 18 Apr 2016 08:51:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A92C320154 for ; Mon, 18 Apr 2016 08:51:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 13C8320122 for ; Mon, 18 Apr 2016 08:51:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751840AbcDRIvu (ORCPT ); Mon, 18 Apr 2016 04:51:50 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:43209 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751402AbcDRIvs (ORCPT ); Mon, 18 Apr 2016 04:51:48 -0400 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u3I8pla9007021 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 18 Apr 2016 08:51:47 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id u3I8pko7005663 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 18 Apr 2016 08:51:47 GMT Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id u3I8peWr001712; Mon, 18 Apr 2016 08:51:46 GMT Received: from arch2.sg.oracle.com (/10.186.101.65) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 18 Apr 2016 01:51:40 -0700 From: Anand Jain To: linux-btrfs@vger.kernel.org Cc: yauhen.kharuzhy@zavadatar.com Subject: [PATCH] btrfs: fix lock dep warning, move scratch dev out of device_list_mutex and uuid_mutex Date: Mon, 18 Apr 2016 16:51:23 +0800 Message-Id: <1460969483-7500-1-git-send-email-anand.jain@oracle.com> X-Mailer: git-send-email 2.7.0 X-Source-IP: userv0021.oracle.com [156.151.31.71] Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When the replace target fails, the target device will be taken out of fs device list, scratch + update_dev_time and freed. However we could do the scratch + update_dev_time and free part after the device has been taken out of device list, so that we don't have to hold the device_list_mutex and uuid_mutex locks. Reported issue: [ 5375.718845] ====================================================== [ 5375.718846] [ INFO: possible circular locking dependency detected ] [ 5375.718849] 4.4.5-scst31x-debug-11+ #40 Not tainted [ 5375.718849] ------------------------------------------------------- [ 5375.718851] btrfs-health/4662 is trying to acquire lock: [ 5375.718861] (sb_writers){.+.+.+}, at: [] __sb_start_write+0xb7/0xf0 [ 5375.718862] [ 5375.718862] but task is already holding lock: [ 5375.718907] (&fs_devs->device_list_mutex){+.+.+.}, at: [] btrfs_destroy_dev_replace_tgtdev+0x3c/0x150 [btrfs] [ 5375.718907] [ 5375.718907] which lock already depends on the new lock. [ 5375.718907] [ 5375.718908] [ 5375.718908] the existing dependency chain (in reverse order) is: [ 5375.718911] [ 5375.718911] -> #3 (&fs_devs->device_list_mutex){+.+.+.}: [ 5375.718917] [] lock_acquire+0xce/0x1e0 [ 5375.718921] [] mutex_lock_nested+0x69/0x3c0 [ 5375.718940] [] btrfs_show_devname+0x36/0x210 [btrfs] [ 5375.718945] [] show_vfsmnt+0x49/0x150 [ 5375.718948] [] m_show+0x17/0x20 [ 5375.718951] [] seq_read+0x2d8/0x3b0 [ 5375.718955] [] __vfs_read+0x28/0xd0 [ 5375.718959] [] vfs_read+0x86/0x130 [ 5375.718962] [] SyS_read+0x49/0xa0 [ 5375.718966] [] entry_SYSCALL_64_fastpath+0x16/0x7a [ 5375.718968] [ 5375.718968] -> #2 (namespace_sem){+++++.}: [ 5375.718971] [] lock_acquire+0xce/0x1e0 [ 5375.718974] [] down_write+0x49/0x80 [ 5375.718977] [] lock_mount+0x43/0x1c0 [ 5375.718979] [] do_add_mount+0x23/0xd0 [ 5375.718982] [] do_mount+0x27b/0xe30 [ 5375.718985] [] SyS_mount+0x8c/0xd0 [ 5375.718988] [] entry_SYSCALL_64_fastpath+0x16/0x7a [ 5375.718991] [ 5375.718991] -> #1 (&sb->s_type->i_mutex_key#5){+.+.+.}: [ 5375.718994] [] lock_acquire+0xce/0x1e0 [ 5375.718996] [] mutex_lock_nested+0x69/0x3c0 [ 5375.719001] [] path_openat+0x468/0x1360 [ 5375.719004] [] do_filp_open+0x7e/0xe0 [ 5375.719007] [] do_sys_open+0x12b/0x210 [ 5375.719010] [] SyS_open+0x1e/0x20 [ 5375.719013] [] entry_SYSCALL_64_fastpath+0x16/0x7a [ 5375.719015] [ 5375.719015] -> #0 (sb_writers){.+.+.+}: [ 5375.719018] [] __lock_acquire+0x17ba/0x1ae0 [ 5375.719021] [] lock_acquire+0xce/0x1e0 [ 5375.719026] [] percpu_down_read+0x4f/0xa0 [ 5375.719028] [] __sb_start_write+0xb7/0xf0 [ 5375.719031] [] mnt_want_write+0x24/0x50 [ 5375.719035] [] path_openat+0xd32/0x1360 [ 5375.719037] [] do_filp_open+0x7e/0xe0 [ 5375.719040] [] file_open_name+0xe4/0x130 [ 5375.719043] [] filp_open+0x33/0x60 [ 5375.719073] [] update_dev_time+0x16/0x40 [btrfs] [ 5375.719099] [] btrfs_scratch_superblocks+0x4e/0x90 [btrfs] [ 5375.719123] [] btrfs_destroy_dev_replace_tgtdev+0x65/0x150 [btrfs] [ 5375.719150] [] btrfs_dev_replace_finishing+0x6b0/0x990 [btrfs] [ 5375.719175] [] btrfs_dev_replace_start+0x33e/0x540 [btrfs] [ 5375.719199] [] btrfs_auto_replace_start+0xf8/0x140 [btrfs] [ 5375.719222] [] health_kthread+0x246/0x490 [btrfs] [ 5375.719225] [] kthread+0xef/0x110 [ 5375.719229] [] ret_from_fork+0x3f/0x70 [ 5375.719230] [ 5375.719230] other info that might help us debug this: [ 5375.719230] [ 5375.719233] Chain exists of: [ 5375.719233] sb_writers --> namespace_sem --> &fs_devs->device_list_mutex [ 5375.719233] [ 5375.719234] Possible unsafe locking scenario: [ 5375.719234] [ 5375.719234] CPU0 CPU1 [ 5375.719235] ---- ---- [ 5375.719236] lock(&fs_devs->device_list_mutex); [ 5375.719238] lock(namespace_sem); [ 5375.719239] lock(&fs_devs->device_list_mutex); [ 5375.719241] lock(sb_writers); [ 5375.719241] [ 5375.719241] *** DEADLOCK *** [ 5375.719241] [ 5375.719243] 4 locks held by btrfs-health/4662: [ 5375.719266] #0: (&fs_info->health_mutex){+.+.+.}, at: [] health_kthread+0x63/0x490 [btrfs] [ 5375.719293] #1: (&fs_info->dev_replace.lock_finishing_cancel_unmount){+.+.+.}, at: [] btrfs_dev_replace_finishing+0x41/0x990 [btrfs] [ 5375.719319] #2: (uuid_mutex){+.+.+.}, at: [] btrfs_destroy_dev_replace_tgtdev+0x20/0x150 [btrfs] [ 5375.719343] #3: (&fs_devs->device_list_mutex){+.+.+.}, at: [] btrfs_destroy_dev_replace_tgtdev+0x3c/0x150 [btrfs] [ 5375.719343] [ 5375.719343] stack backtrace: [ 5375.719347] CPU: 2 PID: 4662 Comm: btrfs-health Not tainted 4.4.5-scst31x-debug-11+ #40 [ 5375.719348] Hardware name: Supermicro SYS-6018R-WTRT/X10DRW-iT, BIOS 1.0c 01/07/2015 [ 5375.719352] 0000000000000000 ffff880856f73880 ffffffff813529e3 ffffffff826182a0 [ 5375.719354] ffffffff8260c090 ffff880856f738c0 ffffffff810d667c ffff880856f73930 [ 5375.719357] ffff880861f32b40 ffff880861f32b68 0000000000000003 0000000000000004 [ 5375.719357] Call Trace: [ 5375.719363] [] dump_stack+0x85/0xc2 [ 5375.719366] [] print_circular_bug+0x1ec/0x260 [ 5375.719369] [] __lock_acquire+0x17ba/0x1ae0 [ 5375.719373] [] ? debug_lockdep_rcu_enabled+0x1d/0x20 [ 5375.719376] [] lock_acquire+0xce/0x1e0 [ 5375.719378] [] ? __sb_start_write+0xb7/0xf0 [ 5375.719383] [] percpu_down_read+0x4f/0xa0 [ 5375.719385] [] ? __sb_start_write+0xb7/0xf0 [ 5375.719387] [] __sb_start_write+0xb7/0xf0 [ 5375.719389] [] mnt_want_write+0x24/0x50 [ 5375.719393] [] path_openat+0xd32/0x1360 [ 5375.719415] [] ? btrfs_congested_fn+0x180/0x180 [btrfs] [ 5375.719418] [] ? debug_lockdep_rcu_enabled+0x1d/0x20 [ 5375.719420] [] do_filp_open+0x7e/0xe0 [ 5375.719423] [] ? rcu_read_lock_sched_held+0x6d/0x80 [ 5375.719426] [] ? kmem_cache_alloc+0x26b/0x5d0 [ 5375.719430] [] ? getname_kernel+0x34/0x120 [ 5375.719433] [] file_open_name+0xe4/0x130 [ 5375.719436] [] filp_open+0x33/0x60 [ 5375.719462] [] update_dev_time+0x16/0x40 [btrfs] [ 5375.719485] [] btrfs_scratch_superblocks+0x4e/0x90 [btrfs] [ 5375.719506] [] btrfs_destroy_dev_replace_tgtdev+0x65/0x150 [btrfs] [ 5375.719530] [] btrfs_dev_replace_finishing+0x6b0/0x990 [btrfs] [ 5375.719554] [] ? btrfs_dev_replace_finishing+0x553/0x990 [btrfs] [ 5375.719576] [] btrfs_dev_replace_start+0x33e/0x540 [btrfs] [ 5375.719598] [] btrfs_auto_replace_start+0xf8/0x140 [btrfs] [ 5375.719621] [] health_kthread+0x246/0x490 [btrfs] [ 5375.719641] [] ? health_kthread+0x138/0x490 [btrfs] [ 5375.719661] [] ? btrfs_congested_fn+0x180/0x180 [btrfs] [ 5375.719663] [] kthread+0xef/0x110 [ 5375.719666] [] ? kthread_create_on_node+0x200/0x200 [ 5375.719669] [] ret_from_fork+0x3f/0x70 [ 5375.719672] [] ? kthread_create_on_node+0x200/0x200 [ 5375.719697] ------------[ cut here ]------------ Signed-off-by: Anand Jain Reported-by: Yauhen Kharuzhy Reviewed-by: David Sterba --- fs/btrfs/volumes.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 4e8936d322d6..051af04836e1 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -2080,10 +2080,9 @@ void btrfs_destroy_dev_replace_tgtdev(struct btrfs_fs_info *fs_info, btrfs_sysfs_rm_device_link(fs_info->fs_devices, tgtdev); - if (tgtdev->bdev) { - btrfs_scratch_superblocks(tgtdev->bdev, tgtdev->name->str); + if (tgtdev->bdev) fs_info->fs_devices->open_devices--; - } + fs_info->fs_devices->num_devices--; next_device = list_entry(fs_info->fs_devices->devices.next, @@ -2094,10 +2093,18 @@ void btrfs_destroy_dev_replace_tgtdev(struct btrfs_fs_info *fs_info, fs_info->fs_devices->latest_bdev = next_device->bdev; list_del_rcu(&tgtdev->dev_list); - call_rcu(&tgtdev->rcu, free_device); - mutex_unlock(&fs_info->fs_devices->device_list_mutex); mutex_unlock(&uuid_mutex); + + /* + * The update_dev_time() with in btrfs_scratch_superblocks() + * may lead to a call to btrfs_show_devname() which will try + * to hold device_list_mutex. And here this device + * is already out of device list, so we don't have to hold + * the device_list_mutex lock. + */ + btrfs_scratch_superblocks(tgtdev->bdev, tgtdev->name->str); + call_rcu(&tgtdev->rcu, free_device); } static int btrfs_find_device_by_path(struct btrfs_root *root, char *device_path,