From patchwork Fri Aug 7 04:01:19 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liu Bo X-Patchwork-Id: 6965001 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id C5843C05AC for ; Fri, 7 Aug 2015 04:01:49 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 6D9A72069A for ; Fri, 7 Aug 2015 04:01:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0C2B620695 for ; Fri, 7 Aug 2015 04:01:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750825AbbHGEBm (ORCPT ); Fri, 7 Aug 2015 00:01:42 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:17609 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750726AbbHGEBk (ORCPT ); Fri, 7 Aug 2015 00:01:40 -0400 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id t7741Zrj015125 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 7 Aug 2015 04:01:35 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.13.8/8.13.8) with ESMTP id t7741Yo4001566 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 7 Aug 2015 04:01:35 GMT Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10]) by aserv0122.oracle.com (8.13.8/8.13.8) with ESMTP id t7741SY5030039; Fri, 7 Aug 2015 04:01:34 GMT Received: from localhost.localdomain (/10.182.228.124) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 06 Aug 2015 21:01:28 -0700 Date: Fri, 7 Aug 2015 12:01:19 +0800 From: Liu Bo To: Elias Probst Cc: linux-btrfs@vger.kernel.org Subject: Re: Lockup in BTRFS_IOC_CLONE/Kernel 4.2.0-rc5 Message-ID: <20150807040119.GA8887@localhost.localdomain> Reply-To: bo.li.liu@oracle.com References: <55C1C915.4070106@eliasprobst.eu> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <55C1C915.4070106@eliasprobst.eu> User-Agent: Mutt/1.5.23 (2014-03-12) X-Source-IP: userv0021.oracle.com [156.151.31.71] Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi, On Wed, Aug 05, 2015 at 10:28:05AM +0200, Elias Probst wrote: > I can reproduce a hard btrfs lockup (process issuing the ioctl() is in > D-state, same goes for btrfs-transacti process) on Kernel 4.2.0-rc5. > > I had the same issue on 4.1, so it's unlikely a regression introduced in > 4.2. > > ## With the following steps, I can reproduce the problem: > > 1. Create a new clean btrfs volume for /var/lib/machines > machinectl set-limit 6G > > 2. Paste this to /tmp/yum.conf > [main] > reposdir=/dev/null > gpgcheck=0 > logfile=/var/log/yum.log > installroot=/var/lib/machines/centos7.1-base > assumeyes=1 > > [base] > name=CentOS 7.1.1503 - x86_64 > baseurl=http://mirror.centos.org/centos/7.1.1503/os/x86_64/ > enabled=1 > > 3. Bootstrap a CentOS 7.1 base image > /usr/bin/yum -c /tmp/yum.conf groupinstall Base > > 4. Start an ephemeral systemd-nspawn container based on 'centos7.1-base' > strace -o /tmp/systemd-nspawn.out -s 500 -f systemd-nspawn -xbD > /var/lib/machines/centos7.1-base/ > > > `systemd-nspawn` will now just hang forever. > I couldn't come up yet with a shorter/more low-level way to reproduce this as I lack quite a bit of btrfs experience. Thank you for reporting this. Could you do 'echo w > /proc/sysrq-trigger' to gather the whole hang call stack? Here's a quick patch that may address your problem, can you give it a shot after getting sysrq-w output? Thanks, -liubo > > ## Results: > > - Last 'strace' lines > 6095 fchown(16, 0, 0) = 0 > 6095 fchmod(16, 0755) = 0 > 6095 utimensat(16, NULL, {{1402362275, 0}, {1438761285, 819041906}}, 0) = 0 > 6095 flistxattr(15, "", 100) = 0 > 6095 getdents(15, /* 3 entries */, 32768) = 80 > 6095 newfstatat(15, "coreutils.mo", {st_mode=S_IFREG|0644, st_size=357263, ...}, AT_SYMLINK_NOFOLLOW) = 0 > 6095 openat(15, "coreutils.mo", O_RDONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = 17 > 6095 openat(16, "coreutils.mo", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, 0644) = 18 > 6095 fstat(18, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 > 6095 ioctl(18, BTRFS_IOC_CLONE > > - call trace in Kernel journal: > Aug 05 10:10:03 moria kernel: INFO: task btrfs-transacti:4175 blocked for more than 120 seconds. > Aug 05 10:10:03 moria kernel: Tainted: G O 4.2.0-rc5 #2 > Aug 05 10:10:03 moria kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Aug 05 10:10:03 moria kernel: btrfs-transacti D ffff8800b13279f8 0 4175 2 0x00080080 > Aug 05 10:10:03 moria kernel: ffff8800b13279f8 ffff88018fd3a380 ffff8800ab4521c0 0000000000000246 > Aug 05 10:10:03 moria kernel: ffff8800b1328000 ffff88018d5c8518 ffff88018debdba0 ffff880232d64990 > Aug 05 10:10:03 moria kernel: 0000000000000197 ffff8800b1327a18 ffffffff86999201 0000000000000000 > Aug 05 10:10:03 moria kernel: Call Trace: > Aug 05 10:10:03 moria kernel: [] schedule+0x74/0x83 > Aug 05 10:10:03 moria kernel: [] btrfs_tree_lock+0xa7/0x1b7 > Aug 05 10:10:03 moria kernel: [] ? wait_woken+0x74/0x74 > Aug 05 10:10:03 moria kernel: [] push_leaf_right+0x9a/0x19f > Aug 05 10:10:03 moria kernel: [] split_leaf+0x100/0x63f > Aug 05 10:10:03 moria kernel: [] ? leaf_space_used+0xbb/0xea > Aug 05 10:10:03 moria kernel: [] ? btrfs_set_lock_blocking_rw+0x52/0x95 > Aug 05 10:10:03 moria kernel: [] btrfs_search_slot+0x76c/0x8b3 > Aug 05 10:10:03 moria kernel: [] btrfs_insert_empty_items+0x58/0xa3 > Aug 05 10:10:03 moria kernel: [] btrfs_insert_delayed_items+0x7f/0x3bb > Aug 05 10:10:03 moria kernel: [] __btrfs_run_delayed_items+0x98/0x1c0 > Aug 05 10:10:03 moria kernel: [] btrfs_run_delayed_items+0xc/0xe > Aug 05 10:10:03 moria kernel: [] btrfs_commit_transaction+0x298/0xb66 > Aug 05 10:10:03 moria kernel: [] ? start_transaction+0x3b2/0x535 > Aug 05 10:10:03 moria kernel: [] transaction_kthread+0x100/0x1d6 > Aug 05 10:10:03 moria kernel: [] ? btrfs_cleanup_transaction+0x49f/0x49f > Aug 05 10:10:03 moria kernel: [] kthread+0xcd/0xd5 > Aug 05 10:10:03 moria kernel: [] ? kthread_create_on_node+0x17d/0x17d > Aug 05 10:10:03 moria kernel: [] ret_from_fork+0x3f/0x70 > Aug 05 10:10:03 moria kernel: [] ? kthread_create_on_node+0x17d/0x17d > Aug 05 10:10:03 moria kernel: INFO: task systemd-nspawn:6095 blocked for more than 120 seconds. > Aug 05 10:10:03 moria kernel: Tainted: G O 4.2.0-rc5 #2 > Aug 05 10:10:03 moria kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Aug 05 10:10:03 moria kernel: systemd-nspawn D ffff88019f3e3668 0 6095 6090 0x00080083 > Aug 05 10:10:03 moria kernel: ffff88019f3e3668 ffffffff86e5d480 ffff88018fd3a380 0000000000000246 > Aug 05 10:10:03 moria kernel: ffff88019f3e4000 ffff88018debdc08 ffff88018fd3a380 ffff88019f3e36b8 > Aug 05 10:10:03 moria kernel: ffff88018fd3a380 ffff88019f3e3688 ffffffff86999201 0000000000000000 > Aug 05 10:10:03 moria kernel: Call Trace: > Aug 05 10:10:03 moria kernel: [] schedule+0x74/0x83 > Aug 05 10:10:03 moria kernel: [] btrfs_tree_read_lock+0xc0/0xea > Aug 05 10:10:03 moria kernel: [] ? wait_woken+0x74/0x74 > Aug 05 10:10:03 moria kernel: [] btrfs_search_old_slot+0x51a/0x810 > Aug 05 10:10:03 moria kernel: [] btrfs_next_old_leaf+0xdf/0x3ce > Aug 05 10:10:03 moria kernel: [] ? ulist_add_merge+0x1b/0x127 > Aug 05 10:10:03 moria kernel: [] __resolve_indirect_refs+0x62a/0x667 > Aug 05 10:10:03 moria kernel: [] ? btrfs_clear_lock_blocking_rw+0x78/0xbe > Aug 05 10:10:03 moria kernel: [] find_parent_nodes+0xaf3/0xfc6 > Aug 05 10:10:03 moria kernel: [] __btrfs_find_all_roots+0x92/0xf0 > Aug 05 10:10:03 moria kernel: [] btrfs_find_all_roots+0x45/0x65 > Aug 05 10:10:03 moria kernel: [] ? btrfs_get_tree_mod_seq+0x2b/0x88 > Aug 05 10:10:03 moria kernel: [] check_ref+0x64/0xc4 > Aug 05 10:10:03 moria kernel: [] btrfs_clone+0x66e/0xb5d > Aug 05 10:10:03 moria kernel: [] btrfs_ioctl_clone+0x48f/0x5bb > Aug 05 10:10:03 moria kernel: [] ? native_sched_clock+0x28/0x77 > Aug 05 10:10:03 moria kernel: [] btrfs_ioctl+0xabc/0x25cb > Aug 05 10:10:03 moria kernel: [] ? __schedule+0x590/0x7f8 > Aug 05 10:10:03 moria kernel: [] ? schedule+0x74/0x83 > Aug 05 10:10:03 moria kernel: [] ? ptrace_stop+0x227/0x236 > Aug 05 10:10:03 moria kernel: [] ? account_system_time+0xf2/0x10f > Aug 05 10:10:03 moria kernel: [] ? ptrace_do_notify+0x7f/0x8c > Aug 05 10:10:03 moria kernel: [] do_vfs_ioctl+0x369/0x423 > Aug 05 10:10:03 moria kernel: [] ? __audit_syscall_entry+0xba/0xdc > Aug 05 10:10:03 moria kernel: [] SyS_ioctl+0x39/0x61 > Aug 05 10:10:03 moria kernel: [] tracesys_phase2+0x84/0x89 > --- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 0770c91..b52bd66 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -3478,6 +3478,22 @@ process_slot: drop_start = new_key.offset; /* + * We need to look up the roots that point at + * this bytenr and see if the new root does. If + * it does not we need to make sure we update + * quotas appropriately. + */ + if (disko && root != BTRFS_I(src)->root && + disko != last_disko) { + no_quota = check_ref(trans, root, + disko); + if (no_quota < 0) { + ret = no_quota; + goto out; + } + } + + /* * 1 - adjusting old extent (we may have to * split it) * 1 - add new extent * 1 - inode update @@ -3544,27 +3560,6 @@ process_slot: btrfs_set_file_extent_num_bytes(leaf, extent, datal); - /* - * We need to look up the roots that point at - * this bytenr and see if the new root does. If - * it does not we need to make sure we update - * quotas appropriately. - */ - if (disko && root != BTRFS_I(src)->root && - disko != last_disko) { - no_quota = check_ref(trans, root, - disko); - if (no_quota < 0) { - btrfs_abort_transaction(trans, - root, - ret); - btrfs_end_transaction(trans, - root); - ret = no_quota; - goto out; - } - } - if (disko) { inode_add_bytes(inode, datal); ret = btrfs_inc_extent_ref(trans, root,