| Message ID | 20170628144348.abvqowzmeveyzssn@merlins.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote: >[cc trimmed] > >On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote: >> Because the output is abnormal, except for the relevant DIR_ITEM and >> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA. >> I wonder if the file system is online when this command is executed? If >> so, please re-execute it offline again; if not, could you apply my >> patches re-check it again? > >The filesystem was offline and I had those 2 patches applied. I am afraid I don't know why the inode item disappers. Besides, if btrfs-debug-tree can't find the inode item, btrfs check shouldn't report this inode item's extent data interrupt. Could you check the disk again? The error output may have changed. > >Marc >-- >"A mouse is a device used to point at the xterm you want to type in" - A.S.R. >Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking >Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
On Thu, Jun 29, 2017 at 09:36:15PM +0800, Lu Fengqi wrote: > On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote: > >[cc trimmed] > > > >On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote: > >> Because the output is abnormal, except for the relevant DIR_ITEM and > >> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA. > >> I wonder if the file system is online when this command is executed? If > >> so, please re-execute it offline again; if not, could you apply my > >> patches re-check it again? > > > >The filesystem was offline and I had those 2 patches applied. > > I am afraid I don't know why the inode item disappers. Besides, if > btrfs-debug-tree can't find the inode item, btrfs check shouldn't report > this inode item's extent data interrupt. Could you check the disk > again? The error output may have changed. I just did but it takes 24H. I just have the results now: gargamel:~# btrfs check --mode lowmem /dev/mapper/dshelf2 Checking filesystem on /dev/mapper/dshelf2 UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede checking extents checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt ERROR: errors found in fs roots found 5544779124736 bytes used, error(s) found total csum bytes: 5344523140 total tree bytes: 71323058176 total fs tree bytes: 59288403968 total extent tree bytes: 5378277376 btree space waste bytes: 10912183048 file data blocks allocated: 7830914256896 referenced 6244104495104 This is looking better, but not 0. Can I ignore these or should we look into them still? Marc
On Thu, Jun 29, 2017 at 08:30:35AM -0700, Marc MERLIN wrote: >On Thu, Jun 29, 2017 at 09:36:15PM +0800, Lu Fengqi wrote: >> On Wed, Jun 28, 2017 at 07:43:48AM -0700, Marc MERLIN wrote: >> >[cc trimmed] >> > >> >On Wed, Jun 28, 2017 at 03:10:27PM +0800, Lu Fengqi wrote: >> >> Because the output is abnormal, except for the relevant DIR_ITEM and >> >> DIR_INDEX, I can't find the above mentiond INODE_ITEM and EXTENT_DATA. >> >> I wonder if the file system is online when this command is executed? If >> >> so, please re-execute it offline again; if not, could you apply my >> >> patches re-check it again? >> > >> >The filesystem was offline and I had those 2 patches applied. >> >> I am afraid I don't know why the inode item disappers. Besides, if >> btrfs-debug-tree can't find the inode item, btrfs check shouldn't report >> this inode item's extent data interrupt. Could you check the disk >> again? The error output may have changed. > >I just did but it takes 24H. I just have the results now: >gargamel:~# btrfs check --mode lowmem /dev/mapper/dshelf2 >Checking filesystem on /dev/mapper/dshelf2 >UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede >checking extents >checking free space cache >cache and super generation don't match, space cache will be invalidated >checking fs roots >ERROR: root 3862 EXTENT_DATA[18170706 4096] interrupt >ERROR: root 3862 EXTENT_DATA[18170706 16384] interrupt >ERROR: root 3862 EXTENT_DATA[18170706 20480] interrupt >ERROR: root 3862 EXTENT_DATA[18170706 135168] interrupt >ERROR: root 3862 EXTENT_DATA[18170706 1048576] interrupt >ERROR: errors found in fs roots >found 5544779124736 bytes used, error(s) found >total csum bytes: 5344523140 >total tree bytes: 71323058176 >total fs tree bytes: 59288403968 >total extent tree bytes: 5378277376 >btree space waste bytes: 10912183048 >file data blocks allocated: 7830914256896 > referenced 6244104495104 > > >This is looking better, but not 0. >Can I ignore these or should we look into them still? > >Marc >-- >"A mouse is a device used to point at the xterm you want to type in" - A.S.R. >Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking >Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 > > Personally, I think since the normal mode didn't report any error related this inode, then these error maybe caused by the bug of lowmem mode and btrfs-debug-tree. At your convenience, would you please give me all items about this inode? I think it can provide some clues regarding the disappearance of inode and the extent interrupt. It can be dumped by this following command: # btrfs-debug-tree /dev/mapper/dshelf2 | grep -C 10 18170706 Please pay attention that, this dump may contain filenames, feel free to mask the filenames. Thank you for your assistance.
I'm still trying to fix my filesystem. It seems to work well enough since the damage is apparently localized, but I'd really want check --repair to actually bring it back to a working state, but now it's crashing This is btrfs tools from git from a few days ago Failed to find [4068943577088, 168, 16384] btrfs unable to find ref byte nr 4068943577088 parent 0 root 4 owner 1 offset 0 Failed to find [5905106075648, 168, 16384] btrfs unable to find ref byte nr 5906282119168 parent 0 root 4 owner 0 offset 1 Failed to find [21037056, 168, 16384] btrfs unable to find ref byte nr 21037056 parent 0 root 3 owner 1 offset 0 Failed to find [21053440, 168, 16384] btrfs unable to find ref byte nr 21053440 parent 0 root 3 owner 0 offset 1 Failed to find [21299200, 168, 16384] btrfs unable to find ref byte nr 21299200 parent 0 root 3 owner 0 offset 1 Failed to find [5523931971584, 168, 16384] btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861 owner 3 offset 0 ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5 btrfs(+0x113cf)[0x5651e60443cf] btrfs(__btrfs_cow_block+0x576)[0x5651e6045848] btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6] btrfs(btrfs_search_slot+0x11df)[0x5651e604969d] btrfs(+0x59184)[0x5651e608c184] btrfs(cmd_check+0x2bd4)[0x5651e60987b3] btrfs(main+0x85)[0x5651e60442c3] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1] btrfs(_start+0x2a)[0x5651e6043e3a] Full log: enabling repair mode Checking filesystem on /dev/mapper/dshelf2 UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede checking extents Fixed 0 roots. checking free space cache cache and super generation don't match, space cache will be invalidated checking fs roots checksum verify failed on 3037243965440 found 179689AF wanted 82B97043 checksum verify failed on 3037243965440 found 179689AF wanted 82B97043 checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09 checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09 checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598 checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598 checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135 checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135 checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7 checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7 checksum verify failed on 3111569391616 found 3C623707 wanted D955D668 checksum verify failed on 3111569391616 found 3C623707 wanted D955D668 checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2 checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2 checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35 checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35 checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339 checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339 checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43 checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43 checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0 checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0 checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10 checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10 checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81 checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81 checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B Csum didn't match The following tree block(s) is corrupted in tree 3861: tree block bytenr: 1710573748224, level: 1, node key: (1073956, 12, 959325) Try to repair the btree for root 3861 Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Csum didn't match Failed to find [4068943577088, 168, 16384] btrfs unable to find ref byte nr 4068943577088 parent 0 root 4 owner 1 offset 0 Failed to find [5905106075648, 168, 16384] btrfs unable to find ref byte nr 5906282119168 parent 0 root 4 owner 0 offset 1 Failed to find [21037056, 168, 16384] btrfs unable to find ref byte nr 21037056 parent 0 root 3 owner 1 offset 0 Failed to find [21053440, 168, 16384] btrfs unable to find ref byte nr 21053440 parent 0 root 3 owner 0 offset 1 Failed to find [21299200, 168, 16384] btrfs unable to find ref byte nr 21299200 parent 0 root 3 owner 0 offset 1 Failed to find [5523931971584, 168, 16384] btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861 owner 3 offset 0 ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5 btrfs(+0x113cf)[0x5651e60443cf] btrfs(__btrfs_cow_block+0x576)[0x5651e6045848] btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6] btrfs(btrfs_search_slot+0x11df)[0x5651e604969d] btrfs(+0x59184)[0x5651e608c184] btrfs(cmd_check+0x2bd4)[0x5651e60987b3] btrfs(main+0x85)[0x5651e60442c3] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1] btrfs(_start+0x2a)[0x5651e6043e3a] Aborted gargamel:~#
On Thu, Jul 06, 2017 at 10:37:18PM -0700, Marc MERLIN wrote: > I'm still trying to fix my filesystem. > It seems to work well enough since the damage is apparently localized, but > I'd really want check --repair to actually bring it back to a working > state, but now it's crashing > > This is btrfs tools from git from a few days ago > > Failed to find [4068943577088, 168, 16384] > btrfs unable to find ref byte nr 4068943577088 parent 0 root 4 owner 1 offset 0 > Failed to find [5905106075648, 168, 16384] > btrfs unable to find ref byte nr 5906282119168 parent 0 root 4 owner 0 offset 1 > Failed to find [21037056, 168, 16384] > btrfs unable to find ref byte nr 21037056 parent 0 root 3 owner 1 offset 0 > Failed to find [21053440, 168, 16384] > btrfs unable to find ref byte nr 21053440 parent 0 root 3 owner 0 offset 1 > Failed to find [21299200, 168, 16384] > btrfs unable to find ref byte nr 21299200 parent 0 root 3 owner 0 offset 1 > Failed to find [5523931971584, 168, 16384] > btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861 owner 3 offset 0 > ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5 > btrfs(+0x113cf)[0x5651e60443cf] > btrfs(__btrfs_cow_block+0x576)[0x5651e6045848] > btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6] > btrfs(btrfs_search_slot+0x11df)[0x5651e604969d] > btrfs(+0x59184)[0x5651e608c184] > btrfs(cmd_check+0x2bd4)[0x5651e60987b3] > btrfs(main+0x85)[0x5651e60442c3] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1] > btrfs(_start+0x2a)[0x5651e6043e3a] Mmmh, never mind, it seems that the software raid suffered yet another double disk failure due to some undermined flakiness in the underlying block device cabling :-/ That would likely explain the failures here. > Full log: > enabling repair mode > Checking filesystem on /dev/mapper/dshelf2 > UUID: 85441c59-ad11-4b25-b1fe-974f9e4acede > checking extents > Fixed 0 roots. > checking free space cache > cache and super generation don't match, space cache will be invalidated > checking fs roots > checksum verify failed on 3037243965440 found 179689AF wanted 82B97043 > checksum verify failed on 3037243965440 found 179689AF wanted 82B97043 > checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F > checksum verify failed on 3037243998208 found 60EA5C5B wanted 0CF5948F > checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E > checksum verify failed on 3037244293120 found 38382803 wanted 39E4F85E > checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C > checksum verify failed on 3037244342272 found E84F1D8F wanted 472DA98C > checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09 > checksum verify failed on 3037244669952 found 2F6E4C0E wanted E00BBF09 > checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA > checksum verify failed on 3037248913408 found CE2E4AEE wanted EF22F9CA > checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC > checksum verify failed on 3037248929792 found C989CB0E wanted E27527BC > checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598 > checksum verify failed on 3037247569920 found 05848C79 wanted EF3D5598 > checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135 > checksum verify failed on 3037247586304 found 9D1E4E39 wanted F1EC8135 > checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D > checksum verify failed on 3037247619072 found BFE40520 wanted 627DB20D > checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC > checksum verify failed on 3037249208320 found A6B5775F wanted B1E6C0FC > checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7 > checksum verify failed on 3037252534272 found 207AD7DF wanted DE72BDF7 > checksum verify failed on 3111569391616 found 3C623707 wanted D955D668 > checksum verify failed on 3111569391616 found 3C623707 wanted D955D668 > checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A > checksum verify failed on 3111569768448 found 0C129F3C wanted C509003A > checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2 > checksum verify failed on 3111569735680 found E94C9D41 wanted 55836DD2 > checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35 > checksum verify failed on 3037253435392 found 8E124EB5 wanted A3291C35 > checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339 > checksum verify failed on 3037253746688 found 2B6A4DCD wanted 4323B339 > checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43 > checksum verify failed on 3111569702912 found 1048610C wanted 9856BB43 > checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF > checksum verify failed on 3111569801216 found CD7AAF82 wanted C1DA44DF > checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE > checksum verify failed on 3037251878912 found 86FB02F3 wanted 728772CE > checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0 > checksum verify failed on 3037252861952 found CFD54426 wanted E91774C0 > checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE > checksum verify failed on 3037255974912 found E3655B7C wanted 8163FDDE > checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10 > checksum verify failed on 3037252927488 found E7AD88A3 wanted F6BA5B10 > checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81 > checksum verify failed on 3037253500928 found 514A55B2 wanted 3611CD81 > checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B > checksum verify failed on 3037256105984 found 41ADA274 wanted 8F7F0A0B > Csum didn't match > The following tree block(s) is corrupted in tree 3861: > tree block bytenr: 1710573748224, level: 1, node key: (1073956, 12, 959325) > Try to repair the btree for root 3861 > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Csum didn't match > Failed to find [4068943577088, 168, 16384] > btrfs unable to find ref byte nr 4068943577088 parent 0 root 4 owner 1 offset 0 > Failed to find [5905106075648, 168, 16384] > btrfs unable to find ref byte nr 5906282119168 parent 0 root 4 owner 0 offset 1 > Failed to find [21037056, 168, 16384] > btrfs unable to find ref byte nr 21037056 parent 0 root 3 owner 1 offset 0 > Failed to find [21053440, 168, 16384] > btrfs unable to find ref byte nr 21053440 parent 0 root 3 owner 0 offset 1 > Failed to find [21299200, 168, 16384] > btrfs unable to find ref byte nr 21299200 parent 0 root 3 owner 0 offset 1 > Failed to find [5523931971584, 168, 16384] > btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861 owner 3 offset 0 > ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5 > btrfs(+0x113cf)[0x5651e60443cf] > btrfs(__btrfs_cow_block+0x576)[0x5651e6045848] > btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6] > btrfs(btrfs_search_slot+0x11df)[0x5651e604969d] > btrfs(+0x59184)[0x5651e608c184] > btrfs(cmd_check+0x2bd4)[0x5651e60987b3] > btrfs(main+0x85)[0x5651e60442c3] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1] > btrfs(_start+0x2a)[0x5651e6043e3a] > Aborted > gargamel:~# > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/
On Thu, Jul 06, 2017 at 10:39:53PM -0700, Marc MERLIN wrote: >On Thu, Jul 06, 2017 at 10:37:18PM -0700, Marc MERLIN wrote: >> I'm still trying to fix my filesystem. >> It seems to work well enough since the damage is apparently localized, but >> I'd really want check --repair to actually bring it back to a working >> state, but now it's crashing I apologise for my late reply. As a colleague left, I have to take over his work recently. >> >> This is btrfs tools from git from a few days ago >> >> Failed to find [4068943577088, 168, 16384] >> btrfs unable to find ref byte nr 4068943577088 parent 0 root 4 owner 1 offset 0 >> Failed to find [5905106075648, 168, 16384] >> btrfs unable to find ref byte nr 5906282119168 parent 0 root 4 owner 0 offset 1 >> Failed to find [21037056, 168, 16384] >> btrfs unable to find ref byte nr 21037056 parent 0 root 3 owner 1 offset 0 >> Failed to find [21053440, 168, 16384] >> btrfs unable to find ref byte nr 21053440 parent 0 root 3 owner 0 offset 1 >> Failed to find [21299200, 168, 16384] >> btrfs unable to find ref byte nr 21299200 parent 0 root 3 owner 0 offset 1 >> Failed to find [5523931971584, 168, 16384] >> btrfs unable to find ref byte nr 5524037566464 parent 0 root 3861 owner 3 offset 0 >> ctree.c:197: update_ref_for_cow: BUG_ON `ret` triggered, value -5 >> btrfs(+0x113cf)[0x5651e60443cf] >> btrfs(__btrfs_cow_block+0x576)[0x5651e6045848] >> btrfs(btrfs_cow_block+0xea)[0x5651e6045dc6] >> btrfs(btrfs_search_slot+0x11df)[0x5651e604969d] >> btrfs(+0x59184)[0x5651e608c184] >> btrfs(cmd_check+0x2bd4)[0x5651e60987b3] >> btrfs(main+0x85)[0x5651e60442c3] >> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f34f523d2b1] >> btrfs(_start+0x2a)[0x5651e6043e3a] > >Mmmh, never mind, it seems that the software raid suffered yet another >double disk failure due to some undermined flakiness in the underlying block >device cabling :-/ >That would likely explain the failures here. I'm sorry for hear this. Which raid level are you using? So could you recover from this double disk failure?
On Fri, Jul 07, 2017 at 05:33:20PM +0800, Lu Fengqi wrote: > I apologise for my late reply. As a colleague left, I have to take over his > work recently. no worries. > >Mmmh, never mind, it seems that the software raid suffered yet another > >double disk failure due to some undermined flakiness in the underlying block > >device cabling :-/ > >That would likely explain the failures here. > > I'm sorry for hear this. Which raid level are you using? So could you recover > from this double disk failure? The disks aren't failed, and the array wasn't being written to. It's just a matter of putting the disks back in the md raid5 array in the right order. Marc
Sigh, This is now the 3rd filesystem I have (on 3 different machines) that is getting corruption of some kind (on 4.11.6). This is starting to look suspicious :-/ Can I fix this filesystem in some other way? gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 enabling repair mode Checking filesystem on /dev/mapper/crypt_bcache2 UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57 checking extents ref mismatch on [14655689654272 16384] extent item 0, found 1 Backref 14655689654272 parent 15455 root 15455 not found in extent tree backpointer mismatch on [14655689654272 16384] owner ref check failed [14655689654272 16384] repair deleting extent record: key 14655689654272 169 1 adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455 Repaired extent references for 14655689654272 root 15455 has a root item with a more recent gen (33682) compared to the found root node (0) ERROR: failed to repair root items: Invalid argument Recreating the filesystem is going to take me a week of work, a lot of if manual, and I'm not feeling very good with doing this since the backup server this is a backup of, is also seeing some hopefully minor) problems too. I really hope there isn't a new corruption problem in 4.11, because when I'm getting corruption on my laptop, my backup server, and the backup of my backup server, I'm starting to run out of redundant backups :( (and I'm not mentioning all the time this is costing me) Marc
+Chris On Sat, Jul 08, 2017 at 09:34:17PM -0700, Marc MERLIN wrote: > gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 > enabling repair mode > Checking filesystem on /dev/mapper/crypt_bcache2 > UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57 > checking extents > ref mismatch on [14655689654272 16384] extent item 0, found 1 > Backref 14655689654272 parent 15455 root 15455 not found in extent tree > backpointer mismatch on [14655689654272 16384] > owner ref check failed [14655689654272 16384] > repair deleting extent record: key 14655689654272 169 1 > adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455 > Repaired extent references for 14655689654272 > root 15455 has a root item with a more recent gen (33682) compared to the found root node (0) > ERROR: failed to repair root items: Invalid argument On this note, getting hit 3 times on 3 different filesystems, that are not badly damaged, but in none of those caess can btrfs check --repair put them in a working state, is really bringing home the problem with lack of proper fsck. I understand that some errors are hard to fix without unknown data loss, but btrfs check --repair should just do what it takes to put the filesystem back into a consistent state, never mind what data is lost. Restoring 10 to 20TB of data is getting old and is not really an acceptable answer as the only way out. I should not have to recreate a filesystem as the only way to bring it back to a working state. Before Duncan tells me my filesystem is too big, and I should keep to very small filesystems so that it's less work for each time btrfs gets corrupted again, and fails again to bring back the filesystem to a usable state after discarding some data, that's just not an acceptable answer long term, and by long term honestly I mean now. I just have data that doesn't segment well and the more small filesystems I make the more time I'm going to waste managing them all and dealing with which one gets full first :( So, whether 4.11 has a corruption problem, or not, please put some resources behind btrfs check --repair, be it the lowmem mode, or not. Thank you Marc
On Sat, Jul 08, 2017 at 09:34:17PM -0700, Marc MERLIN wrote: > Sigh, > > This is now the 3rd filesystem I have (on 3 different machines) that is > getting corruption of some kind (on 4.11.6). > This is starting to look suspicious :-/ > > Can I fix this filesystem in some other way? > gargamel:/var/local/scr/host# btrfs check --repair /dev/mapper/crypt_bcache2 > enabling repair mode > Checking filesystem on /dev/mapper/crypt_bcache2 > UUID: c4e6f9ca-e9a2-43d7-befa-763fc2cd5a57 > checking extents > ref mismatch on [14655689654272 16384] extent item 0, found 1 > Backref 14655689654272 parent 15455 root 15455 not found in extent tree > backpointer mismatch on [14655689654272 16384] > owner ref check failed [14655689654272 16384] > repair deleting extent record: key 14655689654272 169 1 > adding new tree backref on start 14655689654272 len 16384 parent 0 root 15455 > Repaired extent references for 14655689654272 > root 15455 has a root item with a more recent gen (33682) compared to the found root node (0) > ERROR: failed to repair root items: Invalid argument Mmmh, actually to be fair, this was the 2nd run, I didn't scroll back enough and missed the first run (doing too many recoveries at once, I'm getting mixed up). This first run looks like a lot more things happened: http://marc.merlins.org/tmp/btrfs_check_ds5.txt The amount of things that went wrong here are very worrisome, given that there were no issues with those drives and that array has been working for over a year without problems, until I recently upgraded to 4.11 :( Now mind you, despite the 21MB of things that got fixed, I still kind of have the expectation that btrfs check --repairs continues and fixes everything until the filesystem is clean again, just like e2fsck -f would, but I understand that this filesystem somehow got corrupted to a point that it's maybe not that simple to do so. Marc
Hello Marc. Marc MERLIN - 08.07.17, 21:34: > Sigh, > > This is now the 3rd filesystem I have (on 3 different machines) that is > getting corruption of some kind (on 4.11.6). Anyone else getting corruptions with 4.11? I happily switch back to 4.10.17 or even 4.9 if that is the case. I may even do so just from your reports. Well, yes, I will do exactly that. I just switch back for 4.10 for now. Better be safe, than sorry. I know how you feel, Marc. I posted about a corruption on one of my backup harddisks here some time ago that btrfs check --repair wasn´t able to handle. I redid that disk from scratch and it took a long, long time. I agree with you that this has to stop. Before that I will never *ever* recommend this to a customer. Ideally no corruptions in stable kernels, especially when its a .6 at the end of the version number. But if so… then fixable. Other filesystems like Ext4 and XFS can do it… so this should be possible with BTRFS as well. Thanks,
> -----Original Message----- > From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs- > owner@vger.kernel.org] On Behalf Of Martin Steigerwald > Sent: Sunday, 9 July 2017 5:58 PM > To: Marc MERLIN <marc@merlins.org> > Cc: Lu Fengqi <lufq.fnst@cn.fujitsu.com>; Btrfs BTRFS <linux- > btrfs@vger.kernel.org>; David Sterba <dsterba@suse.cz> > Subject: Re: 4.11.6 / more corruption / root 15455 has a root item with a more > recent gen (33682) compared to the found root node (0) > > Hello Marc. > > Marc MERLIN - 08.07.17, 21:34: > > Sigh, > > > > This is now the 3rd filesystem I have (on 3 different machines) that > > is getting corruption of some kind (on 4.11.6). > > Anyone else getting corruptions with 4.11? > > I happily switch back to 4.10.17 or even 4.9 if that is the case. I may even do > so just from your reports. Well, yes, I will do exactly that. I just switch back > for 4.10 for now. Better be safe, than sorry. No corruption for me - I've been on 4.11 since about .2 and everything seems fine. Currently on 4.11.8 Paul.
Paul Jones posted on Sun, 09 Jul 2017 09:16:36 +0000 as excerpted: >> Marc MERLIN - 08.07.17, 21:34: >> > >> > This is now the 3rd filesystem I have (on 3 different machines) that >> > is getting corruption of some kind (on 4.11.6). >> >> Anyone else getting corruptions with 4.11? >> >> I happily switch back to 4.10.17 or even 4.9 if that is the case. I may >> even do so just from your reports. Well, yes, I will do exactly that. I >> just switch back for 4.10 for now. Better be safe, than sorry. > > No corruption for me - I've been on 4.11 since about .2 and everything > seems fine. Currently on 4.11.8 No corruptions here either. 4.12.0 now, previously 4.12-rc5(ish, git), before that 4.11.0. I have however just upgraded to new ssds then wiped and setup the old ones as another backup set, so everything is on brand new filesystems on fast ssds, no possibility of old undetected corruption suddenly triggering problems. Also, all my btrfs are raid1 or dup for checksummed redundancy, and relatively small, the largest now 80 GiB per device, after the upgrade. And my use-case doesn't involve snapshots or subvolumes. So any bug that is most likely on older filesystems, say those without the no-holes feature, for instance, or that doesn't tend to hit raid1 or dup mode, or that is less likely on small filesystems on fast ssds, or that triggers most often with reflinks and thus on filesystems with snapshots, is unlikely to hit me.
Hello Duncan. Duncan - 09.07.17, 11:17: > Paul Jones posted on Sun, 09 Jul 2017 09:16:36 +0000 as excerpted: > >> Marc MERLIN - 08.07.17, 21:34: > >> > This is now the 3rd filesystem I have (on 3 different machines) that > >> > is getting corruption of some kind (on 4.11.6). > >> > >> Anyone else getting corruptions with 4.11? > >> > >> I happily switch back to 4.10.17 or even 4.9 if that is the case. I may > >> even do so just from your reports. Well, yes, I will do exactly that. I > >> just switch back for 4.10 for now. Better be safe, than sorry. > > > > No corruption for me - I've been on 4.11 since about .2 and everything > > seems fine. Currently on 4.11.8 > > No corruptions here either. 4.12.0 now, previously 4.12-rc5(ish, git), > before that 4.11.0. > > I have however just upgraded to new ssds then wiped and setup the old […] > Also, all my btrfs are raid1 or dup for checksummed redundancy, and > relatively small, the largest now 80 GiB per device, after the upgrade. > And my use-case doesn't involve snapshots or subvolumes. > > So any bug that is most likely on older filesystems, say those without > the no-holes feature, for instance, or that doesn't tend to hit raid1 or > dup mode, or that is less likely on small filesystems on fast ssds, or > that triggers most often with reflinks and thus on filesystems with > snapshots, is unlikely to hit me. Hmmm, the BTRFS filesystems on my laptop 3 to 5 or even more years old. I stick with 4.10 for now, I think. The older ones are RAID 1 across two SSDs, the newer one is single device, on one SSD. These filesystems didn´t fail me in years and since 4.5 or 4.6 even the "I search for free space" kernel hang (hung tasks and all that) is gone as well. Thanks,
On 7/9/17, Duncan <1i5t5.duncan@cox.net> wrote: > I have however just upgraded to new ssds then wiped and setup the old > ones as another backup set, so everything is on brand new filesystems on > fast ssds, no possibility of old undetected corruption suddenly > triggering problems. > > Also, all my btrfs are raid1 or dup for checksummed redundancy Do you have any experience/advice/comment regarding dup data on ssds? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Imran Geriskovan posted on Sat, 29 Jul 2017 21:29:46 +0200 as excerpted: > On 7/9/17, Duncan <1i5t5.duncan@cox.net> wrote: >> I have however just upgraded to new ssds then wiped and setup the old >> ones as another backup set, so everything is on brand new filesystems on >> fast ssds, no possibility of old undetected corruption suddenly >> triggering problems. >> >> Also, all my btrfs are raid1 or dup for checksummed redundancy > > Do you have any experience/advice/comment regarding > dup data on ssds? Very good question. =:^) Limited. Most of my btrfs are raid1, with dup only used on the device- respective /boot btrfs (of which there are four, one on each of the two ssds that otherwise form the btrfs raid1 pairs, for each of the working and backup copy pairs -- I can use BIOS to select any of the four to boot), and those are all sub-GiB mixed-bg mode. So all my dup experience is sub-GiB mixed-blockgroup mode. Within that limitation, my only btrfs problem has been that at my initially chosen size of 256 MiB, mkfs.btrfs at least used to create an initial data/metadata chunk of 64 MiB. Remember, this is dup mode, so there's two of them = 128 MiB. Because there's also a system chunk, that means the initial chunk cannot be balanced even with an entirely empty filesystem, because there's not enough space to write a second 64 MiB chunk duped to 128 MiB. Between that and the 256 MiB in dup mode size meaning under 128 MiB usable, and the fact that I routinely run and sometimes need to bisect pre-release kernels, I was routinely running out of space, then cleaning up, but not being able to do a full cleanup without a blow-away and new mkfs.btrfs, because I couldn't balance. When I recently purchased the second pair of (now larger) ssds in ordered to put everything, including the media and backups that were previously still on spinning rust, on ssd, I redid the layout and made the /boots 512 MiB, still mixed-bg dup mode. That seems to have solved the problem, and I can now rebalance the first mkfs.btrfs-created mixed-bg chunk, as it's now small enough that it's less than half the filesystem even when duped. Because it's now 512 MiB, however, I can't say for sure whether the previous problem with mkfs.btrfs creating an initial mixed-bg chunk of a quarter the 256 MiB filesystem size, so in dup mode it can't be balanced because it's half the total filesystem size and with the system chunk as well, the other half is partially used so there's no space to write the balance destination chunks, is fixed, or not. What I can say is that the problem doesn't affect the new 512 MiB size, at least with btrfs-progs 4.11.x, which is what I used to mkfs.btrfs the new layout.
On 7/30/17, Duncan <1i5t5.duncan@cox.net> wrote: >>> Also, all my btrfs are raid1 or dup for checksummed redundancy >> Do you have any experience/advice/comment regarding >> dup data on ssds? > Very good question. =:^) > Limited. Most of my btrfs are raid1, with dup only used on the device- > respective /boot btrfs (of which there are four, one on each of the two > ssds that otherwise form the btrfs raid1 pairs, for each of the working > and backup copy pairs -- I can use BIOS to select any of the four to > boot), and those are all sub-GiB mixed-bg mode. Is this a military or deep space device? ;) > So all my dup experience is sub-GiB mixed-blockgroup mode. > > Within that limitation, my only btrfs problem has been that at my > initially chosen size of 256 MiB, mkfs.btrfs at least used to create an > initial data/metadata chunk of 64 MiB. Remember, this is dup mode, so > there's two of them = 128 MiB. Because there's also a system chunk, that > means the initial chunk cannot be balanced even with an entirely empty > filesystem, because there's not enough space to write a second 64 MiB > chunk duped to 128 MiB. For /boot, I've also tried dup data. But because of combinations of constraints you've mentioned, I totally give-up trying to have a bullet proof /boot as my poor laptop is not mission critical as your device and as I do always have bootable backups and always carry some bootable sdcards. Perhaps that has something to do with me kicking out all systemd, inits, initramfs, mkinitcpio, dracut, etc, etc. Now the init on /boot is a "19 lines" shell script, including lines for keymap, hdparm, crytpsetup. And let's not forget this is possible by a custom kernel, its reliable buddy syslinux. Interestingly my seach for reliability started with "dup data" and ended up here. :) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Imran Geriskovan posted on Sun, 30 Jul 2017 16:54:25 +0200 as excerpted: > On 7/30/17, Duncan <1i5t5.duncan@cox.net> wrote: >>>> Also, all my btrfs are raid1 or dup for checksummed redundancy > >>> Do you have any experience/advice/comment regarding dup data on ssds? > >> Very good question. =:^) > >> Limited. Most of my btrfs are raid1, with dup only used on the device- >> respective /boot btrfs (of which there are four, one on each of the two >> ssds that otherwise form the btrfs raid1 pairs, for each of the working >> and backup copy pairs -- I can use BIOS to select any of the four to >> boot), and those are all sub-GiB mixed-bg mode. > > Is this a military or deep space device? ;) Just happens to have four physical ssds, two pairs, with everything but /boot being paired btrfs raid1. Because I wanted similar partition layout for ease of management, that's a /boot on each one, and because bios can only point to one at a time, that's four separate grub installs [1], each of which is configured to load its own /boot. While four is a bit much, three can certainly be very useful, because it allows a bad grub upgrade to be core-installed to one BIOS-boot partition, while allowing me to fat-finger point it to the wrong /boot on a second device destroying my ability to boot to it as well, and still have a third untouched to boot from. The forth is simply bonus insurance on that, more by accident due to having two pair than because I really needed it. A minimum of three /boots is also quite convenient for my kernel update routine, given I routinely test and sometimes bisect pre-release kernels. The default/working /boot gets the prereleases with a release and stable fallback, the first backup the releases and a stable fallback, and the secondary backups get updated less frequently, generally when I'm doing a / backup cycle as well and there has been either a kernel config or system change substantial enough that I'm no longer confident the older kernels will work correctly with the updated system. Of course the same general testing/release/stable /boot system works well for other related updates, say to the grub menu (I use grub2's bash-like scripting language directly, not the high level stuff which I find too difficult to tweak to my liking) or the initrd, which I attach to the individual kernels at build-time, so a tested kernel selection is a tested initramfs selection as well. > For /boot, I've also tried dup data. > > But because of combinations of constraints you've mentioned, > I totally give-up trying to have a bullet proof /boot as my poor laptop > is not mission critical as your device and as I do always have bootable > backups and always carry some bootable sdcards. When I complained about the 64-MiB default mixed-bg mode chunk size on a 256 MiB filesystem being too big to allow balance in dup mode, a dev answered that in theory chunk sizes are supposed to be limited to 1/8 filesystem size (down to something like a 16 MiB minimum chunk size I think, but might be 8 or 32), but something about my setup, likely the mixed-bg mode as it's less tested, was short-circuiting that, thus the quarter-fs-size 64 MiB chunk sizes, which he agreed didn't make much sense on a 256 MiB filesystem in dup mode. He was able to duplicate the problem, and there seemed no disagreement is was a bug, but I'm not sure if mkfs.btrfs was ever patched to fix it, and of course now with the bigger half-gig filesystem the same 64-MiB initial chunk size is fine. And my other quarter-gig btrfs, log, is raid1, quarter-gig per device, so I'd not see the problem there, mixed-mode or not. (As mentioned in the footnote below, at least in this go-round it's not... more by accident than intent.) Meanwhile, such bugs come with the territory when you're running what might be roughly compared at the commercial software level to late beta or rc level software, or even initial release, pre-service-release-1, level, which I'd argue is a more accurate btrfs comparison at this point. As long as you stay within the known stable areas the danger of it eating your data is relatively small now, but the full feature set isn't there yet, and some of the features that are there are significantly less mature and stable than others. > Perhaps that has something to do with me kicking out all systemd, inits, > initramfs, mkinitcpio, dracut, etc, etc. > > Now the init on /boot is a "19 lines" shell script, including lines for > keymap, hdparm, crytpsetup. And let's not forget this is possible by a > custom kernel, its reliable buddy syslinux. FWIW... I really like grub2, especially it's quite flexible bash-like scripting language (the higher level stuff intended for normal users just isn't flexible enough for me, so I need the scripting language anyway, and once I knew that, the higher level stuff only got in the way) and command line that allow all sorts of stuff like browsing for kernel commandline documentation at the boot prompt that I never imagined possible in a boot manager. And after holding off for awhile, I'm now a cautious adopter and supporter of systemd in general, tho I don't use its solutions for /everything/ and don't like its extremely aggressive feature expansion. And after resisting an initr* for years as unnecessary, I've been a reluctant adopter since a btrfs raid1 root effectively requires it (rootflags=device= doesn't seem to work, for whatever reason, or at least didn't when I initially converted to btrfs, so at least a limited initr* seems the only viable solution for a btrfs raid1 root). And I'm using dracut for that, tho quite cut down from its default, with a monolithic kernel and only installing necessary dracut modules. But particularly after the last dracut update pulled in kmod as a mandatory dep as it now links against its libs, despite my monolithic kernel built without module support, I've been considering similar initr* alternatives, including hand-rolling my own initr* build scripts. Because I'm still not happy having to run an initr* at all, especially since there's more "magic" there than I'm particularly comfortable with since I like to grok the boot and thus potential recovery process better than I do this, and dracut was just the most convenient option at the time. But kmod isn't a /huge/ dep, particularly with the executables and docs install-masked so it's only the library, headers and *.pc config file installed, and the current dracut solution works /reasonably/ well, so finding/creating an alternative isn't particularly high on my priority list, and I'll probably never do it unless dracut suddenly decides some of its other modules are going to need mandatory deps, or something else radically changes the current fragile balance and I really do need that currently lacking initr* grok. > Interestingly my seach for reliability started with "dup data" and ended > up here. :) =:^) --- [1] Grub and partition layout: I install grub-core (i386-pc) to a raw GPT legacy BIOS boot partition. While this only requires a partition size of about a third of a MiB, I use gdisk's default 1 MiB alignment and the first MiB is the GTP and the alignment gap, so this first BIOS boot partition starts at 1 MiB and must be a whole MiB unit in size. Because I wanted plenty of room, however, and wanted additional partitions a minimum of 4 MiB aligned, I configured a 3 MiB BIOS boot partition for grub to use, thus accomplishing that 4 MiB alignment for further partitions. The second partition is a currently unused GPT EFI partition for forward compatibility, 252 MiB in size so further partitions are quarter-GiB aligned. The third partition is the /boot partition we've been discussing, a half GiB in size, thus ending at 3/4 GiB. It's my only btrfs mixed-mode dup in the layout, so a half gig in size but a quarter gig usable. As mentioned, with four physical ssds that's a total of four /boots, each pointed at by the grub-core installation in the first partition on the corresponding ssd. Partition 4 is the log partition, a quarter GiB in size as log rotation keeps typical usage under 50 MiB, but the quarter gig size means it ends on the 1 GiB boundary and further partitions are GiB aligned. In the last layout generation this was a half gig and /boot a quarter gig, but I decided /boot could use the extra quarter gig more than log so I traded sizes. This, like all further partitions, is btrfs raid1. I intended to make it mixed-bg mode, as it was in the previous generation layout, but forgot the mkfs.btrfs switch for that and it no longer defaults to mixed at under a gig, so I got standard mode. Never-the-less, with raid1 instead of dup, and low normal usage, the chunk size is small enough that balance shouldn't be an issue, and if it is I can always blow it away and recreate in mixed mode. All further partitions are gig-aligned btrfs raid1 pair-device, three copies, working/0 and backups 1 and 2, on two separate pairs of ssds. The older pair is 256GB/238GiB with the backup/1 copy, the newer pair is 1TB/931GiB with working/0 and backup/2. The partition size and layout is identical on all four thru the sub-GiB and first copy, with the second copy on the larger pair being a same-sequence same-size repeat of the first, beyond the non-duplicated sub-GiB, of course. So as long as the GPT on one of the four remains intact and bootable, I can easily recreate the other three.
>>>> Do you have any experience/advice/comment regarding dup data on ssds? >>> Very good question. =:^) >> Now the init on /boot is a "19 lines" shell script, including lines for >> keymap, hdparm, crytpsetup. And let's not forget this is possible by a >> custom kernel and its reliable buddy syslinux. > > FWIW... > And I'm using dracut for that, tho quite cut down from its default, with > a monolithic kernel and only installing necessary dracut modules. Just create minimal bootable /boot for running below init. (Your initramfs/rd is a bloated and packaged version of this anyway.) Kick the rest. Since you a have your own kernel you are not far away from it. #!/bin/sh # This is actually busybox ash or hush. Cant remember now. # You may compile/customize your busybox as well. Easy. mount proc /proc -t proc mount sys /sys -t sysfs mount run /run -t tmpfs mkdir /dev/pts /dev/shm /run/lock mount devpts /dev/pts -t devpts & mount shm /dev/shm -t tmpfs & mount -o remount,rw,noatime / & # '&' is for backgrounding/parallel_execution. # Use responsibly double checking its side effects # depending on your setup. hdparm -B 254 /dev/sda & loadkmap < /boot/trq.bkmap cryptsetup -T 10 luksOpen /dev/sdXX sdXX mount /dev/mapper/sdXX /mnt/new_root -t btrfs -o noatime,compress=lzo cd /mnt/new_root mount --move /dev ./dev mount --move /proc ./proc mount --move /sys ./sys mount --move /run ./run pivot_root . boot exec chroot . busybox init # Jump to your real roots init. Whatever it may be. > But particularly after the last dracut update pulled in kmod as a > mandatory dep as it now links against its libs, despite my monolithic > kernel built without module support, I've been considering similar initr* > alternatives, including hand-rolling my own initr* build scripts. > > Because I'm still not happy having to run an initr* at all, especially > since there's more "magic" there than I'm particularly comfortable with > since I like to grok the boot and thus potential recovery process better > than I do this, and dracut was just the most convenient option at the > time. >> Interestingly my seach for reliability started with "dup data" and ended >> up here. :) > =:^) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>: > Hello Marc. > > Marc MERLIN - 08.07.17, 21:34: >> Sigh, >> >> This is now the 3rd filesystem I have (on 3 different machines) that is >> getting corruption of some kind (on 4.11.6). > > Anyone else getting corruptions with 4.11? Yes, a lot. There are at least 3 cases, probably I've missed something. https://www.spinics.net/lists/linux-btrfs/msg67177.html https://www.spinics.net/lists/linux-btrfs/msg67681.html https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275 If an additional debug info is needed, I'm ready to provide it.
On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote: > 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>: > > Hello Marc. > > > > Marc MERLIN - 08.07.17, 21:34: > >> Sigh, > >> > >> This is now the 3rd filesystem I have (on 3 different machines) that is > >> getting corruption of some kind (on 4.11.6). > > > > Anyone else getting corruptions with 4.11? > Yes, a lot. There are at least 3 cases, probably I've missed something. > https://www.spinics.net/lists/linux-btrfs/msg67177.html > https://www.spinics.net/lists/linux-btrfs/msg67681.html > https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275 Indeed. My main server is happy back on 4.9.36 and while my laptop is stuck on 4.11 due to other kernel issues that prevent me from going back to 4.9, it only corrupted a single filesystem so far, and no other ones that I've noticed yet. Hopefully that will hold :-/ Marc
2017-08-01 0:17 GMT+03:00 Marc MERLIN <marc@merlins.org>: > On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote: >> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>: >> > Hello Marc. >> > >> > Marc MERLIN - 08.07.17, 21:34: >> >> Sigh, >> >> >> >> This is now the 3rd filesystem I have (on 3 different machines) that is >> >> getting corruption of some kind (on 4.11.6). >> > >> > Anyone else getting corruptions with 4.11? >> Yes, a lot. There are at least 3 cases, probably I've missed something. >> https://www.spinics.net/lists/linux-btrfs/msg67177.html >> https://www.spinics.net/lists/linux-btrfs/msg67681.html >> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275 > > Indeed. My main server is happy back on 4.9.36 and while my laptop is > stuck on 4.11 due to other kernel issues that prevent me from going back > to 4.9, it only corrupted a single filesystem so far, and no other ones > that I've noticed yet. > Hopefully that will hold :-/ > > Marc > -- > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > Microsoft is to operating systems .... > .... what McDonalds is to gourmet cooking > Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 I want to try mounting and checking FS under Live images with different kernels tomorrow. Today's Fedora Rawhide image seems to be built incorrectly. Can you advice me where to get a fresh live image with 4.12 kernel (it's not important which distro that will be)?
On Mon, Jul 31, 2017 at 2:17 PM, Marc MERLIN <marc@merlins.org> wrote: > On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote: >> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>: >> > Hello Marc. >> > >> > Marc MERLIN - 08.07.17, 21:34: >> >> Sigh, >> >> >> >> This is now the 3rd filesystem I have (on 3 different machines) that is >> >> getting corruption of some kind (on 4.11.6). >> > >> > Anyone else getting corruptions with 4.11? >> Yes, a lot. There are at least 3 cases, probably I've missed something. >> https://www.spinics.net/lists/linux-btrfs/msg67177.html >> https://www.spinics.net/lists/linux-btrfs/msg67681.html >> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275 > > Indeed. My main server is happy back on 4.9.36 and while my laptop is > stuck on 4.11 due to other kernel issues that prevent me from going back > to 4.9, it only corrupted a single filesystem so far, and no other ones > that I've noticed yet. > Hopefully that will hold :-/ > Marc, do you have quotas enabled? IIRC, you're a send/receive user. The combination of quotas and btrfs receive can corrupt your filesystem, as shown by the xfstest I sent to the list a little while ago. -Justin -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Imran Geriskovan posted on Mon, 31 Jul 2017 22:32:39 +0200 as excerpted: >>> Now the init on /boot is a "19 lines" shell script, including lines >>> for keymap, hdparm, crytpsetup. And let's not forget this is possible >>> by a custom kernel and its reliable buddy syslinux. >> >> FWIW... >> And I'm using dracut for that, tho quite cut down from its default, >> with a monolithic kernel and only installing necessary dracut modules. > > Just create minimal bootable /boot for running below init. > (Your initramfs/rd is a bloated and packaged version of this anyway.) > Kick the rest. Since you a have your own kernel you are not far away > from it. Thanks. You just solved my primary problem of needing to take the time to actually research all the steps and in what order I needed to do them, for a hand-rolled script. =:^) Unfortunately, while I've been laid-up the last ~5 days due to a twisted knee and have been spending more time on the lists, etc, and would have loved to spend a day or so testing and setting this up, I'm back to work tomorrow, so I've no idea when I'll actually get to play with this. But meanwhile, I'm saving your message for reference when the time comes. It should be /very/ useful! =:^)
On Mon, Jul 31, 2017 at 03:00:53PM -0700, Justin Maggard wrote: > Marc, do you have quotas enabled? IIRC, you're a send/receive user. > The combination of quotas and btrfs receive can corrupt your > filesystem, as shown by the xfstest I sent to the list a little while > ago. Thanks for checking. I do not use quota given the problems I had with them early on over 2y ago. Marc
On 8/1/17, Duncan <1i5t5.duncan@cox.net> wrote: > Imran Geriskovan posted on Mon, 31 Jul 2017 22:32:39 +0200 as excerpted: >>>> Now the init on /boot is a "19 lines" shell script, including lines >>>> for keymap, hdparm, crytpsetup. And let's not forget this is possible >>>> by a custom kernel and its reliable buddy syslinux. >>> And I'm using dracut for that, tho quite cut down from its default, >>> with a monolithic kernel and only installing necessary dracut modules. >> Just create minimal bootable /boot for running below init. >> (Your initramfs/rd is a bloated and packaged version of this anyway.) >> Kick the rest. Since you a have your own kernel you are not far away >> from it. > Thanks. You just solved my primary problem of needing to take the time > to actually research all the steps and in what order I needed to do them, > for a hand-rolled script. =:^) It's just a minimal one. But it is a good start. For possible extensions extract your initramfs and explore it. Dracut is bloated. Try mkinitcpio. Once your have your self hosting bootmng, kernel, modules, /boot, init, etc chain, you'll be shocked to realize you have been spending so much time for that bullshit while trying to keep them up.. Get to this point in the shortest possible time. Save your precious time. And reclaim your systems reliability. For X, you'll still need udev or eudev. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
2017-08-01 0:39 GMT+03:00 Ivan Sizov <sivan606@gmail.com>: > 2017-08-01 0:17 GMT+03:00 Marc MERLIN <marc@merlins.org>: >> On Tue, Aug 01, 2017 at 12:07:14AM +0300, Ivan Sizov wrote: >>> 2017-07-09 10:57 GMT+03:00 Martin Steigerwald <martin@lichtvoll.de>: >>> > Hello Marc. >>> > >>> > Marc MERLIN - 08.07.17, 21:34: >>> >> Sigh, >>> >> >>> >> This is now the 3rd filesystem I have (on 3 different machines) that is >>> >> getting corruption of some kind (on 4.11.6). >>> > >>> > Anyone else getting corruptions with 4.11? >>> Yes, a lot. There are at least 3 cases, probably I've missed something. >>> https://www.spinics.net/lists/linux-btrfs/msg67177.html >>> https://www.spinics.net/lists/linux-btrfs/msg67681.html >>> https://unix.stackexchange.com/questions/369133/dealing-with-btrfs-ref-backpointer-mismatches-backref-missing/369275 >> >> Indeed. My main server is happy back on 4.9.36 and while my laptop is >> stuck on 4.11 due to other kernel issues that prevent me from going back >> to 4.9, it only corrupted a single filesystem so far, and no other ones >> that I've noticed yet. >> Hopefully that will hold :-/ >> >> Marc >> -- >> "A mouse is a device used to point at the xterm you want to type in" - A.S.R. >> Microsoft is to operating systems .... >> .... what McDonalds is to gourmet cooking >> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 > > I want to try mounting and checking FS under Live images with > different kernels tomorrow. Today's Fedora Rawhide image seems to be > built incorrectly. Can you advice me where to get a fresh live image > with 4.12 kernel (it's not important which distro that will be)? > > -- > Ivan Sizov Mounting problem persists: on 4.13.0 with btrfs-progs v4.11.1 (latest Fedora Rawhide Live) on 4.10.0 with btrfs-progs v4.9.1 (Ubuntu 17.04 Live) on 4.9.0 with btrfs-progs v 4.7.3 (Debian 9 Stretch Live) "btrfs check --readonly" also gives the same output on 4.11, 4.10 and 4.9. Marc, how did you roll back and fix those errors?
From lufq.fnst@cn.fujitsu.com Mon Jun 26 03:37:41 2017 Received: from [59.151.112.132] (port=50126 helo=heian.cn.fujitsu.com) by mail1.merlins.org with esmtp (Exim 4.87 #1) id 1dPROj-0001kT-Tq for <marc@merlins.org>; Mon, 26 Jun 2017 03:37:41 -0700 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="20491848" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 26 Jun 2017 18:37:30 +0800 Received: from G08CNEXCHPEKD02.g08.fujitsu.local (unknown [10.167.33.83]) by cn.fujitsu.com (Postfix) with ESMTP id B3C5047E64D5; Mon, 26 Jun 2017 18:37:30 +0800 (CST) Received: from lufq.5F.lufq.5F (10.167.225.63) by G08CNEXCHPEKD02.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 26 Jun 2017 18:37:32 +0800 From: Lu Fengqi <lufq.fnst@cn.fujitsu.com> To: <linux-btrfs@vger.kernel.org> CC: <marc@merlins.org> Date: Mon, 26 Jun 2017 18:37:25 +0800 Message-ID: <20170626103727.8945-2-lufq.fnst@cn.fujitsu.com> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20170626103727.8945-1-lufq.fnst@cn.fujitsu.com> References: <20170626103727.8945-1-lufq.fnst@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.167.225.63] X-yoursite-MailScanner-ID: B3C5047E64D5.AC56F X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lufq.fnst@cn.fujitsu.com X-Broken-Reverse-DNS: no host name for IP address 59.151.112.132 X-SA-Exim-Connect-IP: 59.151.112.132 X-SA-Exim-Rcpt-To: marc@merlins.org X-SA-Exim-Mail-From: lufq.fnst@cn.fujitsu.com X-Spam-Checker-Version: SpamAssassin 3.4.1-mmrules_20121111 (2015-04-28) on magic.merlins.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=7.0 tests=BAYES_00,GREYLIST_ISWHITE, RDNS_NONE autolearn=ham autolearn_force=no version=3.4.1-mmrules_20121111 X-Spam-Report: * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.8 RDNS_NONE Delivered to internal network by a host with no rDNS * -1.5 GREYLIST_ISWHITE The incoming server has been whitelisted for this * receipient and sender Subject: [PATCH v3 2/4] btrfs-progs: lowmem check: Fix false alert about referencer count mismatch X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on mail1.merlins.org) Status: O Content-Length: 915 Lines: 29 The normal back reference counting doesn't care about the extent referred by the extent data in the shared leaf. The check_extent_data_backref function need to skip the leaf that owner mismatch with the root_id. Reported-by: Marc MERLIN <marc@merlins.org> Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com> --- cmds-check.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/cmds-check.c b/cmds-check.c index 70d2b7f2..f42968cd 100644 --- a/cmds-check.c +++ b/cmds-check.c @@ -10692,7 +10692,8 @@ static int check_extent_data_backref(struct btrfs_fs_info *fs_info, leaf = path.nodes[0]; slot = path.slots[0]; - if (slot >= btrfs_header_nritems(leaf)) + if (slot >= btrfs_header_nritems(leaf) || + btrfs_header_owner(leaf) != root_id) goto next; btrfs_item_key_to_cpu(leaf, &key, slot); if (key.objectid != objectid || key.type != BTRFS_EXTENT_DATA_KEY) -- 2.13.1