diff mbox

[V5] Btrfs: snapshot-aware defrag

Message ID 20130125154236.GA3997@liubo (mailing list archive)
State New, archived
Headers show

Commit Message

Liu Bo Jan. 25, 2013, 3:42 p.m. UTC
On Fri, Jan 25, 2013 at 08:55:58AM -0600, Mitch Harder wrote:
> On Wed, Jan 23, 2013 at 6:52 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> > On Wed, Jan 23, 2013 at 10:05:04AM -0600, Mitch Harder wrote:
> >> On Wed, Jan 23, 2013 at 1:51 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >> > On Tue, Jan 22, 2013 at 11:41:19AM -0600, Mitch Harder wrote:
> >> >> On Thu, Jan 17, 2013 at 8:42 AM, Mitch Harder
> >> >> <mitch.harder@sabayonlinux.org> wrote:
> >> >> > On Wed, Jan 16, 2013 at 6:36 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
> >> >> >> This comes from one of btrfs's project ideas,
> >> >> >> As we defragment files, we break any sharing from other snapshots.
> >> >> >> The balancing code will preserve the sharing, and defrag needs to grow this
> >> >> >> as well.
> >> > [...]
> >> >> >
> >> >> > I've been testing this patch on a 3.7.2 kernel merged with the
> >> >> > for-linus branch for the 3.8_rc kernels, and I'm seeing the following
> >> >> > error:
> >> >> >
> >> >>
> >> >> I've reproduced the error with CONFIG_DEBUG_LIST enabled, which shows
> >> >> some problem with an entry in the list.
> >> >>
> >> >> [59312.260441] ------------[ cut here ]------------
> >> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> >> >> [59312.260458] Hardware name: OptiPlex 745
> >> >> [59312.260461] list_del corruption. next->prev should be
> >> >> ffff88006511c438, but was dead000000200200
> >> >
> >> > LIST_POISON2 -> (000000200200)
> >> > So we can know that the next one is deleted from the list even _earlier_
> >> > than the current one is.
> >> >
> >> > Any other messages before this warning complains?
> >> >
> >>
> >> Just some normal feedback from a metadata balance I had run.
> >
> > Well, these do fit my expectation, since balance also involves with playing with
> > root_list, which may lead to the bad situation.
> >
> >>
> >> [14057.193343] device fsid 28c688c5-7dbd-4071-b271-1bf6726d8835 devid
> >> 1 transid 4 /dev/sda7
> >> [14057.194438] btrfs: force lzo compression
> >> [14057.194446] btrfs: enabling auto defrag
> >> [14057.194449] btrfs: disk space caching is enabled
> >> [14057.194452] btrfs flagging fs with big metadata feature
> >> [14057.194455] btrfs: lzo incompat flag set.
> >> [57508.799193] btrfs: relocating block group 14516486144 flags 4
> >> [57632.178797] btrfs: found 6775 extents
> >> [57633.214701] btrfs: relocating block group 11832131584 flags 4
> >> [57776.400102] btrfs: found 6480 extents
> >> [57777.021175] btrfs: relocating block group 10489954304 flags 4
> >> [57949.182725] btrfs: found 6681 extents
> >> [59312.260441] ------------[ cut here ]------------
> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> >> [59312.260458] Hardware name: OptiPlex 745
> >> ...
> >>
> >> I'm going to try to wrap some debugging around the section of code in
> >> btrfs_clean_old_snapshots() where the dead_roots list is spliced onto
> >> the root list being processed.  The double entry may be slipping in
> >> here.
> >>
> >> 1764         spin_lock(&fs_info->trans_lock);
> >> 1765         list_splice_init(&fs_info->dead_roots, &list);
> >> 1766         spin_unlock(&fs_info->trans_lock);
> >
> > hmm, I don't think there is anything wrong in this code.  But you can
> > give it a shot anyway :)
> >
> 
> I've changed up my reproducer to try some things that may hit the
> issue quicker and more reliably.
> 
> It gave me a slightly different set of warnings in dmesg, which seem
> to suggest issues in the dead_root list.

Great!  Many thanks for nail it down, we really shouldn't iput()
after btrfs_iget().

Could you please try this(remove iput()) and see if it gets us rid of
the trouble?

thanks,
liubo

 


> 
> [43925.656065] device fsid a8f6fadb-3022-4c01-b369-f1f3f638c052 devid
> 1 transid 310 /dev/sda7
> [43925.658062] btrfs: force lzo compression
> [43925.658072] btrfs: enabling auto defrag
> [43925.658075] btrfs: disk space caching is enabled
> [43925.658078] btrfs: lzo incompat flag set.
> [44503.421293] btrfs: unlinked 1 orphans
> [44898.287365] btrfs: unlinked 1 orphans
> [45080.641383] btrfs: unlinked 1 orphans
> [45250.063773] btrfs: unlinked 1 orphans
> [46223.387355] btrfs: unlinked 1 orphans
> [46476.473944] btrfs: unlinked 1 orphans
> [46499.665615] btrfs: unlinked 1 orphans
> [46769.785454] ------------[ cut here ]------------
> [46769.785471] WARNING: at lib/list_debug.c:36 __list_add+0x9d/0xba()
> [46769.785474] Hardware name: OptiPlex 745
> [46769.785478] list_add double add: new=ffff880050c27c38,
> prev=ffff880078f3e720, next=ffff880050c27c38.
> [46769.785480] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46769.785537] Pid: 18291, comm: btrfs-endio-wri Not tainted 3.7.4-sad-v1+ #3
> [46769.785539] Call Trace:
> [46769.785549]  [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46769.785553]  [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46769.785558]  [<ffffffff8120987b>] __list_add+0x9d/0xba
> [46769.785586]  [<ffffffffa0179dd6>] btrfs_add_dead_root+0x42/0x56 [btrfs]
> [46769.785603]  [<ffffffffa0187b67>] btrfs_destroy_inode+0x227/0x25b [btrfs]
> [46769.785611]  [<ffffffff8111393a>] destroy_inode+0x3b/0x54
> [46769.785615]  [<ffffffff81113a9c>] evict+0x149/0x151
> [46769.785619]  [<ffffffff81114322>] iput+0x12c/0x135
> [46769.785636]  [<ffffffffa018455f>] relink_extent_backref+0x669/0x6af [btrfs]
> [46769.785642]  [<ffffffff815e9849>] ? __slab_free+0x17c/0x21b
> [46769.785658]  [<ffffffffa0184d15>] ?
> btrfs_finish_ordered_io+0x770/0x827 [btrfs]
> [46769.785674]  [<ffffffffa0184ce5>] btrfs_finish_ordered_io+0x740/0x827 [btrfs]
> [46769.785691]  [<ffffffffa0184de1>] finish_ordered_fn+0x15/0x17 [btrfs]
> [46769.785706]  [<ffffffffa019e5c9>] worker_loop+0x14c/0x493 [btrfs]
> [46769.785722]  [<ffffffffa019e47d>] ? btrfs_queue_worker+0x258/0x258 [btrfs]
> [46769.785728]  [<ffffffff8104c750>] kthread+0xba/0xc2
> [46769.785732]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46769.785737]  [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46769.785741]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46769.785745] ---[ end trace 7528086f91b151b5 ]---
> [46799.053062] ------------[ cut here ]------------
> [46799.053078] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
> [46799.053082] Hardware name: OptiPlex 745
> [46799.053087] list_del corruption. next->prev should be
> ffff880050c27c38, but was ffff8800057fde38
> [46799.053090] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46799.053163] Pid: 18210, comm: btrfs-cleaner Tainted: G        W
> 3.7.4-sad-v1+ #3
> [46799.053166] Call Trace:
> [46799.053180]  [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46799.053184]  [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46799.053190]  [<ffffffff810ab4e9>] ? __trace_bprintk+0x48/0x4a
> [46799.053194]  [<ffffffff812097a5>] __list_del_entry+0x8d/0x98
> [46799.053198]  [<ffffffff812097be>] list_del+0xe/0x2e
> [46799.053220]  [<ffffffffa017b2f5>]
> btrfs_clean_old_snapshots+0xed/0x150 [btrfs]
> [46799.053235]  [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46799.053249]  [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46799.053254]  [<ffffffff8104c750>] kthread+0xba/0xc2
> [46799.053259]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46799.053264]  [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46799.053269]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46799.053272] ---[ end trace 7528086f91b151b6 ]---
> [46811.162649] ------------[ cut here ]------------
> [46811.162665] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
> [46811.162669] Hardware name: OptiPlex 745
> [46811.162674] list_del corruption, ffff880050c27c38->next is
> LIST_POISON1 (dead000000100100)
> [46811.162678] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46811.162750] Pid: 18210, comm: btrfs-cleaner Tainted: G        W
> 3.7.4-sad-v1+ #3
> [46811.162754] Call Trace:
> [46811.162764]  [<ffffffff81030586>] warn_slowpath_common+0x83/0x9b
> [46811.162771]  [<ffffffff81030641>] warn_slowpath_fmt+0x46/0x48
> [46811.162779]  [<ffffffff810ab4e9>] ? __trace_bprintk+0x48/0x4a
> [46811.162785]  [<ffffffff812097a5>] __list_del_entry+0x8d/0x98
> [46811.162791]  [<ffffffff812097be>] list_del+0xe/0x2e
> [46811.162820]  [<ffffffffa017b2f5>]
> btrfs_clean_old_snapshots+0xed/0x150 [btrfs]
> [46811.162841]  [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46811.162862]  [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46811.162869]  [<ffffffff8104c750>] kthread+0xba/0xc2
> [46811.162875]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.162882]  [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46811.162888]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.162892] ---[ end trace 7528086f91b151b7 ]---
> [46811.162904] BUG: unable to handle kernel paging request at 0000000047c5a000
> [46811.163003] IP: [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003] PGD 0
> [46811.163003] Oops: 0000 [#1] SMP
> [46811.163003] Modules linked in: ipv6 snd_hda_codec_analog
> snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer
> tg3 sr_mod snd i2c_i801 ppdev parport_pc iTCO_wdt iTCO_vendor_support
> lpc_ich pcspkr parport floppy serio_raw microcode ablk_helper cryptd
> lrw xts gf128mul aes_x86_64 sha256_generic fuse xfs nfs lockd sunrpc
> reiserfs btrfs zlib_deflate ext4 jbd2 ext3 jbd ext2 mbcache sl811_hcd
> hid_generic xhci_hcd ohci_hcd uhci_hcd ehci_hcd
> [46811.163003] CPU 0
> [46811.163003] Pid: 18210, comm: btrfs-cleaner Tainted: G        W
> 3.7.4-sad-v1+ #3 Dell Inc.                 OptiPlex 745
>  /0WF810
> [46811.163003] RIP: 0010:[<ffffffffa017b30b>]  [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003] RSP: 0018:ffff8800057fde38  EFLAGS: 00010296
> [46811.163003] RAX: 0000000047c5a000 RBX: ffff880050c27800 RCX: 0000000000000008
> [46811.163003] RDX: 0000000000000000 RSI: ffff8800057fdd70 RDI: ffff880050c27c70
> [46811.163003] RBP: ffff8800057fde78 R08: 0000000000000000 R09: 0000000000000283
> [46811.163003] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880000000000
> [46811.163003] R13: 0000160000000000 R14: ffff8800057fde38 R15: ffff880050c27c38
> [46811.163003] FS:  0000000000000000(0000) GS:ffff88007f200000(0000)
> knlGS:0000000000000000
> [46811.163003] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [46811.163003] CR2: 0000000047c5a000 CR3: 000000003f270000 CR4: 00000000000007f0
> [46811.163003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [46811.163003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [46811.163003] Process btrfs-cleaner (pid: 18210, threadinfo
> ffff8800057fc000, task ffff88007c030d40)
> [46811.163003] Stack:
> [46811.163003]  ffff880050c27c38 ffff88001f488438 ffff8800057fde48
> ffff88002d15b800
> [46811.163003]  ffff88007c030d40 ffff88007c030d40 ffff88007c030d40
> 0000000000000000
> [46811.163003]  ffff8800057fdeb8 ffffffffa0173d7d ffff88002d15b800
> 0000000000000000
> [46811.163003] Call Trace:
> [46811.163003]  [<ffffffffa0173d7d>] cleaner_kthread+0x5a/0xe6 [btrfs]
> [46811.163003]  [<ffffffffa0173d23>] ? transaction_kthread+0x1a0/0x1a0 [btrfs]
> [46811.163003]  [<ffffffff8104c750>] kthread+0xba/0xc2
> [46811.163003]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.163003]  [<ffffffff815f301c>] ret_from_fork+0x7c/0xb0
> [46811.163003]  [<ffffffff8104c696>] ? kthread_freezable_should_stop+0x52/0x52
> [46811.163003] Code: c7 c7 d5 b2 17 a0 31 c0 e8 b4 01 f3 e0 4c 89 ff
> e8 bb e4 08 e1 48 89 df e8 f2 f5 03 00 49 8b 87 c8 fb ff ff 48 8b 80
> 50 01 00 00 <48> 8b 00 4c 01 e8 48 c1 f8 06 48 c1 e0 0c 42 0f b6 44 20
> 3f 31
> [46811.163003] RIP  [<ffffffffa017b30b>]
> btrfs_clean_old_snapshots+0x103/0x150 [btrfs]
> [46811.163003]  RSP <ffff8800057fde38>
> [46811.163003] CR2: 0000000047c5a000
> [46811.238512] ---[ end trace 7528086f91b151b8 ]---
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Mitch Harder Jan. 25, 2013, 6:16 p.m. UTC | #1
On Fri, Jan 25, 2013 at 9:42 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
> On Fri, Jan 25, 2013 at 08:55:58AM -0600, Mitch Harder wrote:
>> On Wed, Jan 23, 2013 at 6:52 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>> > On Wed, Jan 23, 2013 at 10:05:04AM -0600, Mitch Harder wrote:
>> >> On Wed, Jan 23, 2013 at 1:51 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
>> >> > On Tue, Jan 22, 2013 at 11:41:19AM -0600, Mitch Harder wrote:
>> >> >> On Thu, Jan 17, 2013 at 8:42 AM, Mitch Harder
>> >> >> <mitch.harder@sabayonlinux.org> wrote:
>> >> >> > On Wed, Jan 16, 2013 at 6:36 AM, Liu Bo <bo.li.liu@oracle.com> wrote:
>> >> >> >> This comes from one of btrfs's project ideas,
>> >> >> >> As we defragment files, we break any sharing from other snapshots.
>> >> >> >> The balancing code will preserve the sharing, and defrag needs to grow this
>> >> >> >> as well.
>> >> > [...]
>> >> >> >
>> >> >> > I've been testing this patch on a 3.7.2 kernel merged with the
>> >> >> > for-linus branch for the 3.8_rc kernels, and I'm seeing the following
>> >> >> > error:
>> >> >> >
>> >> >>
>> >> >> I've reproduced the error with CONFIG_DEBUG_LIST enabled, which shows
>> >> >> some problem with an entry in the list.
>> >> >>
>> >> >> [59312.260441] ------------[ cut here ]------------
>> >> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
>> >> >> [59312.260458] Hardware name: OptiPlex 745
>> >> >> [59312.260461] list_del corruption. next->prev should be
>> >> >> ffff88006511c438, but was dead000000200200
>> >> >
>> >> > LIST_POISON2 -> (000000200200)
>> >> > So we can know that the next one is deleted from the list even _earlier_
>> >> > than the current one is.
>> >> >
>> >> > Any other messages before this warning complains?
>> >> >
>> >>
>> >> Just some normal feedback from a metadata balance I had run.
>> >
>> > Well, these do fit my expectation, since balance also involves with playing with
>> > root_list, which may lead to the bad situation.
>> >
>> >>
>> >> [14057.193343] device fsid 28c688c5-7dbd-4071-b271-1bf6726d8835 devid
>> >> 1 transid 4 /dev/sda7
>> >> [14057.194438] btrfs: force lzo compression
>> >> [14057.194446] btrfs: enabling auto defrag
>> >> [14057.194449] btrfs: disk space caching is enabled
>> >> [14057.194452] btrfs flagging fs with big metadata feature
>> >> [14057.194455] btrfs: lzo incompat flag set.
>> >> [57508.799193] btrfs: relocating block group 14516486144 flags 4
>> >> [57632.178797] btrfs: found 6775 extents
>> >> [57633.214701] btrfs: relocating block group 11832131584 flags 4
>> >> [57776.400102] btrfs: found 6480 extents
>> >> [57777.021175] btrfs: relocating block group 10489954304 flags 4
>> >> [57949.182725] btrfs: found 6681 extents
>> >> [59312.260441] ------------[ cut here ]------------
>> >> [59312.260454] WARNING: at lib/list_debug.c:62 __list_del_entry+0x8d/0x98()
>> >> [59312.260458] Hardware name: OptiPlex 745
>> >> ...
>> >>
>> >> I'm going to try to wrap some debugging around the section of code in
>> >> btrfs_clean_old_snapshots() where the dead_roots list is spliced onto
>> >> the root list being processed.  The double entry may be slipping in
>> >> here.
>> >>
>> >> 1764         spin_lock(&fs_info->trans_lock);
>> >> 1765         list_splice_init(&fs_info->dead_roots, &list);
>> >> 1766         spin_unlock(&fs_info->trans_lock);
>> >
>> > hmm, I don't think there is anything wrong in this code.  But you can
>> > give it a shot anyway :)
>> >
>>
>> I've changed up my reproducer to try some things that may hit the
>> issue quicker and more reliably.
>>
>> It gave me a slightly different set of warnings in dmesg, which seem
>> to suggest issues in the dead_root list.
>
> Great!  Many thanks for nail it down, we really shouldn't iput()
> after btrfs_iget().
>
> Could you please try this(remove iput()) and see if it gets us rid of
> the trouble?
>
> thanks,
> liubo
>
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 1683f48..c7a0fb7 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -2337,7 +2337,6 @@ out_free_path:
>  out_unlock:
>         unlock_extent_cached(&BTRFS_I(inode)->io_tree, lock_start,
> lock_end,
>                              &cached, GFP_NOFS);
> -       iput(inode);
>         return ret;
>  }
>

With this patch, the cleaner never runs to delete the old roots.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 1683f48..c7a0fb7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -2337,7 +2337,6 @@  out_free_path:
 out_unlock:
 	unlock_extent_cached(&BTRFS_I(inode)->io_tree, lock_start,
lock_end,
 			     &cached, GFP_NOFS);
-	iput(inode);
 	return ret;
 }