[0/3] btrfs-progs: Add check and repair for invalid inode generation
mbox series

Message ID 20190924081120.6283-1-wqu@suse.com
Headers show
Series
  • btrfs-progs: Add check and repair for invalid inode generation
Related show

Message

Qu Wenruo Sept. 24, 2019, 8:11 a.m. UTC
We have at least two user reports about bad inode generation makes
kernel reject the fs.

According to the creation time, the inode is created by some 2014
kernel.
And the generation member of INODE_ITEM is not updated (unlike the
transid member) so the error persists until latest tree-checker detects.

Even the situation can be fixed by reverting back to older kernel and
copying the offending dir/file to another inode and delete the offending
one, it still should be done by btrfs-progs.

This patchset adds such check and repair ability to btrfs-check, with a
simple test image.

Qu Wenruo (3):
  btrfs-progs: check/lowmem: Add check and repair for invalid inode
    generation
  btrfs-progs: check/original: Add check and repair for invalid inode
    generation
  btrfs-progs: fsck-tests: Add test image for invalid inode generation
    repair

 check/main.c                                  |  50 +++++++++++-
 check/mode-lowmem.c                           |  76 ++++++++++++++++++
 check/mode-original.h                         |   1 +
 .../.lowmem_repairable                        |   0
 .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
 5 files changed, 126 insertions(+), 1 deletion(-)
 create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
 create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz

Comments

Ferry Toth Oct. 18, 2019, 8:32 p.m. UTC | #1
Op 24-09-2019 om 10:11 schreef Qu Wenruo:
> We have at least two user reports about bad inode generation makes
> kernel reject the fs.

May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so 
kernel went from 5.0 -> 5.3 (but I was using 4.15 too).

Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@

In initramfs I can try to mount but get something like
btrfs critical corrupt leaf invalid inode generation open_ctree failed

Booting old kernel works just as before, no errors.

> According to the creation time, the inode is created by some 2014
> kernel.

How do I get the creation time?

> And the generation member of INODE_ITEM is not updated (unlike the
> transid member) so the error persists until latest tree-checker detects.
> 
> Even the situation can be fixed by reverting back to older kernel and
> copying the offending dir/file to another inode and delete the offending
> one, it still should be done by btrfs-progs.
> 
How to find the offending dir/file from the command line manually?

> This patchset adds such check and repair ability to btrfs-check, with a
> simple test image.
> 
> Qu Wenruo (3):
>    btrfs-progs: check/lowmem: Add check and repair for invalid inode
>      generation
>    btrfs-progs: check/original: Add check and repair for invalid inode
>      generation
>    btrfs-progs: fsck-tests: Add test image for invalid inode generation
>      repair
> 
>   check/main.c                                  |  50 +++++++++++-
>   check/mode-lowmem.c                           |  76 ++++++++++++++++++
>   check/mode-original.h                         |   1 +
>   .../.lowmem_repairable                        |   0
>   .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>   5 files changed, 126 insertions(+), 1 deletion(-)
>   create mode 100644 tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>   create mode 100644 tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>
Qu Wenruo Oct. 18, 2019, 11:50 p.m. UTC | #2
On 2019/10/19 上午4:32, Ferry Toth wrote:
> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>> We have at least two user reports about bad inode generation makes
>> kernel reject the fs.
> 
> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
> 
> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
> 
> In initramfs I can try to mount but get something like
> btrfs critical corrupt leaf invalid inode generation open_ctree failed
> 
> Booting old kernel works just as before, no errors.
> 
>> According to the creation time, the inode is created by some 2014
>> kernel.
> 
> How do I get the creation time?

# btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
> 
>> And the generation member of INODE_ITEM is not updated (unlike the
>> transid member) so the error persists until latest tree-checker detects.
>>
>> Even the situation can be fixed by reverting back to older kernel and
>> copying the offending dir/file to another inode and delete the offending
>> one, it still should be done by btrfs-progs.
>>
> How to find the offending dir/file from the command line manually?

# find <mount point> -inum <inode number>

Thanks,
Qu

> 
>> This patchset adds such check and repair ability to btrfs-check, with a
>> simple test image.
>>
>> Qu Wenruo (3):
>>    btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>      generation
>>    btrfs-progs: check/original: Add check and repair for invalid inode
>>      generation
>>    btrfs-progs: fsck-tests: Add test image for invalid inode generation
>>      repair
>>
>>   check/main.c                                  |  50 +++++++++++-
>>   check/mode-lowmem.c                           |  76 ++++++++++++++++++
>>   check/mode-original.h                         |   1 +
>>   .../.lowmem_repairable                        |   0
>>   .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>   5 files changed, 126 insertions(+), 1 deletion(-)
>>   create mode 100644
>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>   create mode 100644
>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>
>
Ferry Toth Oct. 19, 2019, 4:24 p.m. UTC | #3
Hi,

Op 19-10-2019 om 01:50 schreef Qu WenRuo:
> 
> 
> On 2019/10/19 上午4:32, Ferry Toth wrote:
>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>> We have at least two user reports about bad inode generation makes
>>> kernel reject the fs.
>>
>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>
>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
>>
>> In initramfs I can try to mount but get something like
>> btrfs critical corrupt leaf invalid inode generation open_ctree failed
>>
>> Booting old kernel works just as before, no errors.
>>
>>> According to the creation time, the inode is created by some 2014
>>> kernel.
>>
>> How do I get the creation time?
> 
> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>

I just went back to the office to reboot to 5.3 and check the creation 
times and found they were 2013 - 2014.

>>
>>> And the generation member of INODE_ITEM is not updated (unlike the
>>> transid member) so the error persists until latest tree-checker detects.
>>>
>>> Even the situation can be fixed by reverting back to older kernel and
>>> copying the offending dir/file to another inode and delete the offending
>>> one, it still should be done by btrfs-progs.
>>>
>> How to find the offending dir/file from the command line manually?
> 
> # find <mount point> -inum <inode number>

This works, thanks.

But appears unpractical. After fix 2 files and reboot, I found 4 more, 
then 16, then I gave up.

> Thanks,
> Qu
> 
>>
>>> This patchset adds such check and repair ability to btrfs-check, with a
>>> simple test image.
>>>
>>> Qu Wenruo (3):
>>>     btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>>       generation
>>>     btrfs-progs: check/original: Add check and repair for invalid inode
>>>       generation
>>>     btrfs-progs: fsck-tests: Add test image for invalid inode generation
>>>       repair
>>>
>>>    check/main.c                                  |  50 +++++++++++-
>>>    check/mode-lowmem.c                           |  76 ++++++++++++++++++
>>>    check/mode-original.h                         |   1 +
>>>    .../.lowmem_repairable                        |   0
>>>    .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>>    5 files changed, 126 insertions(+), 1 deletion(-)
>>>    create mode 100644
>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>    create mode 100644
>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>
>>

I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my 
mounted rootfs with linux 5.0 and captured the log (~1800 lines 209 
errors).

I'm not sure if using the v5.0 kernel and/or checking mounted distorts 
the results? Else I'm going to need a live usb with a v5.3 kernel and 
v5.3 btrfs-progs.

If you like I can share the log. Let me know.

This issue can potentially cause a lot of grief. Our company server runs 
Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with 
~100 snapshots. I guess the problematic inodes need to be fixed on each 
snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?

Do I understand correctly that this FTB is caused by more strict 
checking of the fs by the kernel, while the tools to fix the detected 
corruptions are not yet released?
Qu Wenruo Oct. 20, 2019, 12:26 a.m. UTC | #4
On 2019/10/20 上午12:24, Ferry Toth wrote:
> Hi,
> 
> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>
>>
>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>> We have at least two user reports about bad inode generation makes
>>>> kernel reject the fs.
>>>
>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>
>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
>>>
>>> In initramfs I can try to mount but get something like
>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed
>>>
>>> Booting old kernel works just as before, no errors.
>>>
>>>> According to the creation time, the inode is created by some 2014
>>>> kernel.
>>>
>>> How do I get the creation time?
>>
>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
> 
> I just went back to the office to reboot to 5.3 and check the creation
> times and found they were 2013 - 2014.
> 
>>>
>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>> transid member) so the error persists until latest tree-checker
>>>> detects.
>>>>
>>>> Even the situation can be fixed by reverting back to older kernel and
>>>> copying the offending dir/file to another inode and delete the
>>>> offending
>>>> one, it still should be done by btrfs-progs.
>>>>
>>> How to find the offending dir/file from the command line manually?
>>
>> # find <mount point> -inum <inode number>
> 
> This works, thanks.
> 
> But appears unpractical. After fix 2 files and reboot, I found 4 more,
> then 16, then I gave up.
> 
>> Thanks,
>> Qu
>>
>>>
>>>> This patchset adds such check and repair ability to btrfs-check, with a
>>>> simple test image.
>>>>
>>>> Qu Wenruo (3):
>>>>     btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>>>       generation
>>>>     btrfs-progs: check/original: Add check and repair for invalid inode
>>>>       generation
>>>>     btrfs-progs: fsck-tests: Add test image for invalid inode
>>>> generation
>>>>       repair
>>>>
>>>>    check/main.c                                  |  50 +++++++++++-
>>>>    check/mode-lowmem.c                           |  76
>>>> ++++++++++++++++++
>>>>    check/mode-original.h                         |   1 +
>>>>    .../.lowmem_repairable                        |   0
>>>>    .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>>>    5 files changed, 126 insertions(+), 1 deletion(-)
>>>>    create mode 100644
>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>    create mode 100644
>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>
>>>
> 
> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
> errors).

It's really not recommended to run btrfs check, especially repair on the
mounted fs, unless it's RO.

A new transaction from kernel can easily screw up the repaired fs.
> 
> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
> the results? Else I'm going to need a live usb with a v5.3 kernel and
> v5.3 btrfs-progs.
> 
> If you like I can share the log. Let me know.
> 
> This issue can potentially cause a lot of grief. Our company server runs
> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
> ~100 snapshots. I guess the problematic inodes need to be fixed on each
> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?

Yes.

> 
> Do I understand correctly that this FTB is caused by more strict
> checking of the fs by the kernel, while the tools to fix the detected
> corruptions are not yet released?

Yes.

Thanks,
Qu
Qu Wenruo Oct. 20, 2019, 12:51 a.m. UTC | #5
On 2019/10/20 上午8:26, Qu Wenruo wrote:
> 
> 
> On 2019/10/20 上午12:24, Ferry Toth wrote:
>> Hi,
>>
>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>
>>>
>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>> We have at least two user reports about bad inode generation makes
>>>>> kernel reject the fs.
>>>>
>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>
>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
>>>>
>>>> In initramfs I can try to mount but get something like
>>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed
>>>>
>>>> Booting old kernel works just as before, no errors.
>>>>
>>>>> According to the creation time, the inode is created by some 2014
>>>>> kernel.
>>>>
>>>> How do I get the creation time?
>>>
>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>
>> I just went back to the office to reboot to 5.3 and check the creation
>> times and found they were 2013 - 2014.
>>
>>>>
>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>> transid member) so the error persists until latest tree-checker
>>>>> detects.
>>>>>
>>>>> Even the situation can be fixed by reverting back to older kernel and
>>>>> copying the offending dir/file to another inode and delete the
>>>>> offending
>>>>> one, it still should be done by btrfs-progs.
>>>>>
>>>> How to find the offending dir/file from the command line manually?
>>>
>>> # find <mount point> -inum <inode number>
>>
>> This works, thanks.
>>
>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>> then 16, then I gave up.

Another solution is use "find" to locate the files with creation time
before 2015, and copy them to a new file, then replace the old file with
the new file.

It would be much safer than btrfs check --repair.

Thanks,
Qu


>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>>> This patchset adds such check and repair ability to btrfs-check, with a
>>>>> simple test image.
>>>>>
>>>>> Qu Wenruo (3):
>>>>>     btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>>>>       generation
>>>>>     btrfs-progs: check/original: Add check and repair for invalid inode
>>>>>       generation
>>>>>     btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>> generation
>>>>>       repair
>>>>>
>>>>>    check/main.c                                  |  50 +++++++++++-
>>>>>    check/mode-lowmem.c                           |  76
>>>>> ++++++++++++++++++
>>>>>    check/mode-original.h                         |   1 +
>>>>>    .../.lowmem_repairable                        |   0
>>>>>    .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>>>>    5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>    create mode 100644
>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>    create mode 100644
>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>
>>>>
>>
>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>> errors).
> 
> It's really not recommended to run btrfs check, especially repair on the
> mounted fs, unless it's RO.
> 
> A new transaction from kernel can easily screw up the repaired fs.
>>
>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>> v5.3 btrfs-progs.
>>
>> If you like I can share the log. Let me know.
>>
>> This issue can potentially cause a lot of grief. Our company server runs
>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
> 
> Yes.
> 
>>
>> Do I understand correctly that this FTB is caused by more strict
>> checking of the fs by the kernel, while the tools to fix the detected
>> corruptions are not yet released?
> 
> Yes.
> 
> Thanks,
> Qu
>
Ferry Toth Oct. 20, 2019, 11:50 a.m. UTC | #6
Op 20-10-2019 om 02:26 schreef Qu Wenruo:
> 
> 
> On 2019/10/20 上午12:24, Ferry Toth wrote:
>> Hi,
>>
>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>
>>>
>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>> We have at least two user reports about bad inode generation makes
>>>>> kernel reject the fs.
>>>>
>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>
>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
>>>>
>>>> In initramfs I can try to mount but get something like
>>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed
>>>>
>>>> Booting old kernel works just as before, no errors.
>>>>
>>>>> According to the creation time, the inode is created by some 2014
>>>>> kernel.
>>>>
>>>> How do I get the creation time?
>>>
>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>
>> I just went back to the office to reboot to 5.3 and check the creation
>> times and found they were 2013 - 2014.
>>
>>>>
>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>> transid member) so the error persists until latest tree-checker
>>>>> detects.
>>>>>
>>>>> Even the situation can be fixed by reverting back to older kernel and
>>>>> copying the offending dir/file to another inode and delete the
>>>>> offending
>>>>> one, it still should be done by btrfs-progs.
>>>>>
>>>> How to find the offending dir/file from the command line manually?
>>>
>>> # find <mount point> -inum <inode number>
>>
>> This works, thanks.
>>
>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>> then 16, then I gave up.
>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>>> This patchset adds such check and repair ability to btrfs-check, with a
>>>>> simple test image.
>>>>>
>>>>> Qu Wenruo (3):
>>>>>      btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>>>>        generation
>>>>>      btrfs-progs: check/original: Add check and repair for invalid inode
>>>>>        generation
>>>>>      btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>> generation
>>>>>        repair
>>>>>
>>>>>     check/main.c                                  |  50 +++++++++++-
>>>>>     check/mode-lowmem.c                           |  76
>>>>> ++++++++++++++++++
>>>>>     check/mode-original.h                         |   1 +
>>>>>     .../.lowmem_repairable                        |   0
>>>>>     .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>>>>     5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>     create mode 100644
>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>     create mode 100644
>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>
>>>>
>>
>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>> errors).
> 
> It's really not recommended to run btrfs check, especially repair on the
> mounted fs, unless it's RO.

Yes, I know. As the fs is mounted btrfs refuses to repair it.
So I only forced to skip the mount check, but no repair has been done, 
only the check.

> A new transaction from kernel can easily screw up the repaired fs.
>>
>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>> v5.3 btrfs-progs.
>>
>> If you like I can share the log. Let me know.
>>
>> This issue can potentially cause a lot of grief. Our company server runs
>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
> 
> Yes.
> 
>>
>> Do I understand correctly that this FTB is caused by more strict
>> checking of the fs by the kernel, while the tools to fix the detected
>> corruptions are not yet released?
> 
> Yes.
> 
> Thanks,
> Qu
>
Ferry Toth Oct. 20, 2019, 1:04 p.m. UTC | #7
Op 20-10-2019 om 02:51 schreef Qu Wenruo:
> 
> 
> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>
>>
>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>> Hi,
>>>
>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>
>>>>
>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>> We have at least two user reports about bad inode generation makes
>>>>>> kernel reject the fs.
>>>>>
>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>
>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and / on /@
>>>>>
>>>>> In initramfs I can try to mount but get something like
>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree failed
>>>>>
>>>>> Booting old kernel works just as before, no errors.
>>>>>
>>>>>> According to the creation time, the inode is created by some 2014
>>>>>> kernel.
>>>>>
>>>>> How do I get the creation time?
>>>>
>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>>
>>> I just went back to the office to reboot to 5.3 and check the creation
>>> times and found they were 2013 - 2014.
>>>
>>>>>
>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>> transid member) so the error persists until latest tree-checker
>>>>>> detects.
>>>>>>
>>>>>> Even the situation can be fixed by reverting back to older kernel and
>>>>>> copying the offending dir/file to another inode and delete the
>>>>>> offending
>>>>>> one, it still should be done by btrfs-progs.
>>>>>>
>>>>> How to find the offending dir/file from the command line manually?
>>>>
>>>> # find <mount point> -inum <inode number>
>>>
>>> This works, thanks.
>>>
>>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>>> then 16, then I gave up.
> 
> Another solution is use "find" to locate the files with creation time
> before 2015, and copy them to a new file, then replace the old file with
> the new file.

Hmm. But how do I "find" by creation time (otime)? Do you have a 
suggestion for this?

> It would be much safer than btrfs check --repair.
> 
> Thanks,
> Qu
> 
> 
>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>>> This patchset adds such check and repair ability to btrfs-check, with a
>>>>>> simple test image.
>>>>>>
>>>>>> Qu Wenruo (3):
>>>>>>      btrfs-progs: check/lowmem: Add check and repair for invalid inode
>>>>>>        generation
>>>>>>      btrfs-progs: check/original: Add check and repair for invalid inode
>>>>>>        generation
>>>>>>      btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>> generation
>>>>>>        repair
>>>>>>
>>>>>>     check/main.c                                  |  50 +++++++++++-
>>>>>>     check/mode-lowmem.c                           |  76
>>>>>> ++++++++++++++++++
>>>>>>     check/mode-original.h                         |   1 +
>>>>>>     .../.lowmem_repairable                        |   0
>>>>>>     .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012 bytes
>>>>>>     5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>     create mode 100644
>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>     create mode 100644
>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>
>>>>>
>>>
>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>> errors).
>>
>> It's really not recommended to run btrfs check, especially repair on the
>> mounted fs, unless it's RO.
>>
>> A new transaction from kernel can easily screw up the repaired fs.
>>>
>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>> v5.3 btrfs-progs.
>>>
>>> If you like I can share the log. Let me know.
>>>
>>> This issue can potentially cause a lot of grief. Our company server runs
>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel ~5.6)?
>>
>> Yes.
>>
>>>
>>> Do I understand correctly that this FTB is caused by more strict
>>> checking of the fs by the kernel, while the tools to fix the detected
>>> corruptions are not yet released?
>>
>> Yes.
>>
>> Thanks,
>> Qu
>>
>
Qu Wenruo Oct. 20, 2019, 1:15 p.m. UTC | #8
On 2019/10/20 下午9:04, Ferry Toth wrote:
> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>
>>
>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>
>>>
>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>> Hi,
>>>>
>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>
>>>>>
>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>> kernel reject the fs.
>>>>>>
>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>
>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>> on /@
>>>>>>
>>>>>> In initramfs I can try to mount but get something like
>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>> failed
>>>>>>
>>>>>> Booting old kernel works just as before, no errors.
>>>>>>
>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>> kernel.
>>>>>>
>>>>>> How do I get the creation time?
>>>>>
>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>>>
>>>> I just went back to the office to reboot to 5.3 and check the creation
>>>> times and found they were 2013 - 2014.
>>>>
>>>>>>
>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>> detects.
>>>>>>>
>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>> and
>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>> offending
>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>
>>>>>> How to find the offending dir/file from the command line manually?
>>>>>
>>>>> # find <mount point> -inum <inode number>
>>>>
>>>> This works, thanks.
>>>>
>>>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>>>> then 16, then I gave up.
>>
>> Another solution is use "find" to locate the files with creation time
>> before 2015, and copy them to a new file, then replace the old file with
>> the new file.
> 
> Hmm. But how do I "find" by creation time (otime)? Do you have a
> suggestion for this?

$ touch -t 201501010000 /tmp/sample
$ find <mnt> -not -cnewer /tmp/sample

If you want, you can add -exec to that find, but I'd only add that after
confirming the execution load is verified.

Thanks,
Qu

> 
>> It would be much safer than btrfs check --repair.
>>
>> Thanks,
>> Qu
>>
>>
>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>>
>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>> with a
>>>>>>> simple test image.
>>>>>>>
>>>>>>> Qu Wenruo (3):
>>>>>>>      btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>> inode
>>>>>>>        generation
>>>>>>>      btrfs-progs: check/original: Add check and repair for
>>>>>>> invalid inode
>>>>>>>        generation
>>>>>>>      btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>> generation
>>>>>>>        repair
>>>>>>>
>>>>>>>     check/main.c                                  |  50 +++++++++++-
>>>>>>>     check/mode-lowmem.c                           |  76
>>>>>>> ++++++++++++++++++
>>>>>>>     check/mode-original.h                         |   1 +
>>>>>>>     .../.lowmem_repairable                        |   0
>>>>>>>     .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>> bytes
>>>>>>>     5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>     create mode 100644
>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>     create mode 100644
>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>> errors).
>>>
>>> It's really not recommended to run btrfs check, especially repair on the
>>> mounted fs, unless it's RO.
>>>
>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>
>>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>> v5.3 btrfs-progs.
>>>>
>>>> If you like I can share the log. Let me know.
>>>>
>>>> This issue can potentially cause a lot of grief. Our company server
>>>> runs
>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>> ~5.6)?
>>>
>>> Yes.
>>>
>>>>
>>>> Do I understand correctly that this FTB is caused by more strict
>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>> corruptions are not yet released?
>>>
>>> Yes.
>>>
>>> Thanks,
>>> Qu
>>>
>>
>
Ferry Toth Oct. 20, 2019, 1:29 p.m. UTC | #9
Op 20-10-2019 om 15:15 schreef Qu WenRuo:
> 
> 
> On 2019/10/20 下午9:04, Ferry Toth wrote:
>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>
>>>
>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>> Hi,
>>>>>
>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>
>>>>>>
>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>>> kernel reject the fs.
>>>>>>>
>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>
>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>> on /@
>>>>>>>
>>>>>>> In initramfs I can try to mount but get something like
>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>> failed
>>>>>>>
>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>
>>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>>> kernel.
>>>>>>>
>>>>>>> How do I get the creation time?
>>>>>>
>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your device>
>>>>>
>>>>> I just went back to the office to reboot to 5.3 and check the creation
>>>>> times and found they were 2013 - 2014.
>>>>>
>>>>>>>
>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>> detects.
>>>>>>>>
>>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>>> and
>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>> offending
>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>
>>>>>>> How to find the offending dir/file from the command line manually?
>>>>>>
>>>>>> # find <mount point> -inum <inode number>
>>>>>
>>>>> This works, thanks.
>>>>>
>>>>> But appears unpractical. After fix 2 files and reboot, I found 4 more,
>>>>> then 16, then I gave up.
>>>
>>> Another solution is use "find" to locate the files with creation time
>>> before 2015, and copy them to a new file, then replace the old file with
>>> the new file.
>>
>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>> suggestion for this?
> 
> $ touch -t 201501010000 /tmp/sample
> $ find <mnt> -not -cnewer /tmp/sample

AFAIK this compares file modified date with status changed date. So, no 
search for creation date.

And stat /tmp/sample (sorry dutch lang output):

ferry@ferry-quad:~$ stat /tmp/sample
   Bestand: /tmp/sample
   Grootte: 0            Blokken: 0            IO-blok: 4096   leeg 
normaal bestand
Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   ferry)
Toegang:   2015-01-01 00:00:00.000000000 +0100
Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
Veranderd: 2019-10-20 15:20:50.366163766 +0200
Ontstaan:  -


> If you want, you can add -exec to that find, but I'd only add that after
> confirming the execution load is verified.
> 
> Thanks,
> Qu
> 
>>
>>> It would be much safer than btrfs check --repair.
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>>
>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>> with a
>>>>>>>> simple test image.
>>>>>>>>
>>>>>>>> Qu Wenruo (3):
>>>>>>>>       btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>>> inode
>>>>>>>>         generation
>>>>>>>>       btrfs-progs: check/original: Add check and repair for
>>>>>>>> invalid inode
>>>>>>>>         generation
>>>>>>>>       btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>> generation
>>>>>>>>         repair
>>>>>>>>
>>>>>>>>      check/main.c                                  |  50 +++++++++++-
>>>>>>>>      check/mode-lowmem.c                           |  76
>>>>>>>> ++++++++++++++++++
>>>>>>>>      check/mode-original.h                         |   1 +
>>>>>>>>      .../.lowmem_repairable                        |   0
>>>>>>>>      .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>>> bytes
>>>>>>>>      5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>      create mode 100644
>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>      create mode 100644
>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>> errors).
>>>>
>>>> It's really not recommended to run btrfs check, especially repair on the
>>>> mounted fs, unless it's RO.
>>>>
>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>
>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted distorts
>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>>> v5.3 btrfs-progs.
>>>>>
>>>>> If you like I can share the log. Let me know.
>>>>>
>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>> runs
>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on each
>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>> ~5.6)?
>>>>
>>>> Yes.
>>>>
>>>>>
>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>>> corruptions are not yet released?
>>>>
>>>> Yes.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>
>>
Qu Wenruo Oct. 20, 2019, 2:11 p.m. UTC | #10
On 2019/10/20 下午9:29, Ferry Toth wrote:
> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>
>>
>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>
>>>>
>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>
>>>>>>>
>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>>>> kernel reject the fs.
>>>>>>>>
>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>
>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>> on /@
>>>>>>>>
>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>> failed
>>>>>>>>
>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>
>>>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>>>> kernel.
>>>>>>>>
>>>>>>>> How do I get the creation time?
>>>>>>>
>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>> device>
>>>>>>
>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>> creation
>>>>>> times and found they were 2013 - 2014.
>>>>>>
>>>>>>>>
>>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>> detects.
>>>>>>>>>
>>>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>>>> and
>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>> offending
>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>
>>>>>>>> How to find the offending dir/file from the command line manually?
>>>>>>>
>>>>>>> # find <mount point> -inum <inode number>
>>>>>>
>>>>>> This works, thanks.
>>>>>>
>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>> more,
>>>>>> then 16, then I gave up.
>>>>
>>>> Another solution is use "find" to locate the files with creation time
>>>> before 2015, and copy them to a new file, then replace the old file
>>>> with
>>>> the new file.
>>>
>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>> suggestion for this?
>>
>> $ touch -t 201501010000 /tmp/sample
>> $ find <mnt> -not -cnewer /tmp/sample
> 
> AFAIK this compares file modified date with status changed date. So, no
> search for creation date.
> 
> And stat /tmp/sample (sorry dutch lang output):
> 
> ferry@ferry-quad:~$ stat /tmp/sample
>   Bestand: /tmp/sample
>   Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
> normaal bestand
> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   ferry)
> Toegang:   2015-01-01 00:00:00.000000000 +0100
> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
> Veranderd: 2019-10-20 15:20:50.366163766 +0200
> Ontstaan:  -

My bad, always got confused by o/a/c/mtime, as c really looks like *c*
reation, so I always got confused between ctime and otime.

Then considering not all fs supports otime, find doesn't support that.
I guess it's only possible by other tools....

BTW, did you find any patterns in those existing offending inodes?
I guess it would be faster than finding a tool supporting otime search.

Thanks,
Qu

> 
> 
>> If you want, you can add -exec to that find, but I'd only add that after
>> confirming the execution load is verified.
>>
>> Thanks,
>> Qu
>>
>>>
>>>> It would be much safer than btrfs check --repair.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>>>
>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>> with a
>>>>>>>>> simple test image.
>>>>>>>>>
>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>       btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>>>> inode
>>>>>>>>>         generation
>>>>>>>>>       btrfs-progs: check/original: Add check and repair for
>>>>>>>>> invalid inode
>>>>>>>>>         generation
>>>>>>>>>       btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>> generation
>>>>>>>>>         repair
>>>>>>>>>
>>>>>>>>>      check/main.c                                  |  50
>>>>>>>>> +++++++++++-
>>>>>>>>>      check/mode-lowmem.c                           |  76
>>>>>>>>> ++++++++++++++++++
>>>>>>>>>      check/mode-original.h                         |   1 +
>>>>>>>>>      .../.lowmem_repairable                        |   0
>>>>>>>>>      .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>>>> bytes
>>>>>>>>>      5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>      create mode 100644
>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>      create mode 100644
>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>> errors).
>>>>>
>>>>> It's really not recommended to run btrfs check, especially repair
>>>>> on the
>>>>> mounted fs, unless it's RO.
>>>>>
>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>
>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>> distorts
>>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>>>> v5.3 btrfs-progs.
>>>>>>
>>>>>> If you like I can share the log. Let me know.
>>>>>>
>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>> runs
>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>> each
>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>> ~5.6)?
>>>>>
>>>>> Yes.
>>>>>
>>>>>>
>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>>>> corruptions are not yet released?
>>>>>
>>>>> Yes.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>
>>>
>
Ferry Toth Oct. 20, 2019, 2:24 p.m. UTC | #11
Op 20-10-2019 om 16:11 schreef Qu Wenruo:
> 
> 
> On 2019/10/20 下午9:29, Ferry Toth wrote:
>> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>>
>>>
>>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>>
>>>>>
>>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>>> We have at least two user reports about bad inode generation makes
>>>>>>>>>> kernel reject the fs.
>>>>>>>>>
>>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 19.10 so
>>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>>
>>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>>> on /@
>>>>>>>>>
>>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>>> failed
>>>>>>>>>
>>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>>
>>>>>>>>>> According to the creation time, the inode is created by some 2014
>>>>>>>>>> kernel.
>>>>>>>>>
>>>>>>>>> How do I get the creation time?
>>>>>>>>
>>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>>> device>
>>>>>>>
>>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>>> creation
>>>>>>> times and found they were 2013 - 2014.
>>>>>>>
>>>>>>>>>
>>>>>>>>>> And the generation member of INODE_ITEM is not updated (unlike the
>>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>>> detects.
>>>>>>>>>>
>>>>>>>>>> Even the situation can be fixed by reverting back to older kernel
>>>>>>>>>> and
>>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>>> offending
>>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>>
>>>>>>>>> How to find the offending dir/file from the command line manually?
>>>>>>>>
>>>>>>>> # find <mount point> -inum <inode number>
>>>>>>>
>>>>>>> This works, thanks.
>>>>>>>
>>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>>> more,
>>>>>>> then 16, then I gave up.
>>>>>
>>>>> Another solution is use "find" to locate the files with creation time
>>>>> before 2015, and copy them to a new file, then replace the old file
>>>>> with
>>>>> the new file.
>>>>
>>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>>> suggestion for this?
>>>
>>> $ touch -t 201501010000 /tmp/sample
>>> $ find <mnt> -not -cnewer /tmp/sample
>>
>> AFAIK this compares file modified date with status changed date. So, no
>> search for creation date.
>>
>> And stat /tmp/sample (sorry dutch lang output):
>>
>> ferry@ferry-quad:~$ stat /tmp/sample
>>    Bestand: /tmp/sample
>>    Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
>> normaal bestand
>> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
>> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   ferry)
>> Toegang:   2015-01-01 00:00:00.000000000 +0100
>> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
>> Veranderd: 2019-10-20 15:20:50.366163766 +0200
>> Ontstaan:  -
> 
> My bad, always got confused by o/a/c/mtime, as c really looks like *c*
> reation, so I always got confused between ctime and otime.
> 
> Then considering not all fs supports otime, find doesn't support that.
> I guess it's only possible by other tools....

New stat will support it, but not in Ubuntu 19.10. We did find this:
https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c

and are trying to work out a script that will gzip all files with 
creation < 2015. Then we can unzip again and overwrite.

> BTW, did you find any patterns in those existing offending inodes?
> I guess it would be faster than finding a tool supporting otime search.

I didn't see any logic. A mix of logs, cached files, journal files etc. 
+ a .kde directory in the /

> Thanks,
> Qu
> 
>>
>>
>>> If you want, you can add -exec to that find, but I'd only add that after
>>> confirming the execution load is verified.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>>> It would be much safer than btrfs check --repair.
>>>>>
>>>>> Thanks,
>>>>> Qu
>>>>>
>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Qu
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>>> with a
>>>>>>>>>> simple test image.
>>>>>>>>>>
>>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>>        btrfs-progs: check/lowmem: Add check and repair for invalid
>>>>>>>>>> inode
>>>>>>>>>>          generation
>>>>>>>>>>        btrfs-progs: check/original: Add check and repair for
>>>>>>>>>> invalid inode
>>>>>>>>>>          generation
>>>>>>>>>>        btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>>> generation
>>>>>>>>>>          repair
>>>>>>>>>>
>>>>>>>>>>       check/main.c                                  |  50
>>>>>>>>>> +++++++++++-
>>>>>>>>>>       check/mode-lowmem.c                           |  76
>>>>>>>>>> ++++++++++++++++++
>>>>>>>>>>       check/mode-original.h                         |   1 +
>>>>>>>>>>       .../.lowmem_repairable                        |   0
>>>>>>>>>>       .../bad_inode_geneartion.img.xz               | Bin 0 -> 2012
>>>>>>>>>> bytes
>>>>>>>>>>       5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>>       create mode 100644
>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>>       create mode 100644
>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>>> errors).
>>>>>>
>>>>>> It's really not recommended to run btrfs check, especially repair
>>>>>> on the
>>>>>> mounted fs, unless it's RO.
>>>>>>
>>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>>
>>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>>> distorts
>>>>>>> the results? Else I'm going to need a live usb with a v5.3 kernel and
>>>>>>> v5.3 btrfs-progs.
>>>>>>>
>>>>>>> If you like I can share the log. Let me know.
>>>>>>>
>>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>>> runs
>>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs with
>>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>>> each
>>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>>> ~5.6)?
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>>
>>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>>> checking of the fs by the kernel, while the tools to fix the detected
>>>>>>> corruptions are not yet released?
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>
>>>>
>>
>
Ferry Toth Oct. 21, 2019, 4:01 p.m. UTC | #12
Op 20-10-2019 om 16:24 schreef Ferry Toth:
> Op 20-10-2019 om 16:11 schreef Qu Wenruo:
>>
>>
>> On 2019/10/20 下午9:29, Ferry Toth wrote:
>>> Op 20-10-2019 om 15:15 schreef Qu WenRuo:
>>>>
>>>>
>>>> On 2019/10/20 下午9:04, Ferry Toth wrote:
>>>>> Op 20-10-2019 om 02:51 schreef Qu Wenruo:
>>>>>>
>>>>>>
>>>>>> On 2019/10/20 上午8:26, Qu Wenruo wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2019/10/20 上午12:24, Ferry Toth wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Op 19-10-2019 om 01:50 schreef Qu WenRuo:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2019/10/19 上午4:32, Ferry Toth wrote:
>>>>>>>>>> Op 24-09-2019 om 10:11 schreef Qu Wenruo:
>>>>>>>>>>> We have at least two user reports about bad inode generation 
>>>>>>>>>>> makes
>>>>>>>>>>> kernel reject the fs.
>>>>>>>>>>
>>>>>>>>>> May I add my report? I just upgraded Ubuntu from 19.04 -> 
>>>>>>>>>> 19.10 so
>>>>>>>>>> kernel went from 5.0 -> 5.3 (but I was using 4.15 too).
>>>>>>>>>>
>>>>>>>>>> Booting 5.3 leaves me in initramfs as I have /boot on @boot and /
>>>>>>>>>> on /@
>>>>>>>>>>
>>>>>>>>>> In initramfs I can try to mount but get something like
>>>>>>>>>> btrfs critical corrupt leaf invalid inode generation open_ctree
>>>>>>>>>> failed
>>>>>>>>>>
>>>>>>>>>> Booting old kernel works just as before, no errors.
>>>>>>>>>>
>>>>>>>>>>> According to the creation time, the inode is created by some 
>>>>>>>>>>> 2014
>>>>>>>>>>> kernel.
>>>>>>>>>>
>>>>>>>>>> How do I get the creation time?
>>>>>>>>>
>>>>>>>>> # btrfs ins dump-tree -b <the bytenr reported by kernel> <your
>>>>>>>>> device>
>>>>>>>>
>>>>>>>> I just went back to the office to reboot to 5.3 and check the
>>>>>>>> creation
>>>>>>>> times and found they were 2013 - 2014.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> And the generation member of INODE_ITEM is not updated 
>>>>>>>>>>> (unlike the
>>>>>>>>>>> transid member) so the error persists until latest tree-checker
>>>>>>>>>>> detects.
>>>>>>>>>>>
>>>>>>>>>>> Even the situation can be fixed by reverting back to older 
>>>>>>>>>>> kernel
>>>>>>>>>>> and
>>>>>>>>>>> copying the offending dir/file to another inode and delete the
>>>>>>>>>>> offending
>>>>>>>>>>> one, it still should be done by btrfs-progs.
>>>>>>>>>>>
>>>>>>>>>> How to find the offending dir/file from the command line 
>>>>>>>>>> manually?
>>>>>>>>>
>>>>>>>>> # find <mount point> -inum <inode number>
>>>>>>>>
>>>>>>>> This works, thanks.
>>>>>>>>
>>>>>>>> But appears unpractical. After fix 2 files and reboot, I found 4
>>>>>>>> more,
>>>>>>>> then 16, then I gave up.
>>>>>>
>>>>>> Another solution is use "find" to locate the files with creation time
>>>>>> before 2015, and copy them to a new file, then replace the old file
>>>>>> with
>>>>>> the new file.
>>>>>
>>>>> Hmm. But how do I "find" by creation time (otime)? Do you have a
>>>>> suggestion for this?
>>>>
>>>> $ touch -t 201501010000 /tmp/sample
>>>> $ find <mnt> -not -cnewer /tmp/sample
>>>
>>> AFAIK this compares file modified date with status changed date. So, no
>>> search for creation date.
>>>
>>> And stat /tmp/sample (sorry dutch lang output):
>>>
>>> ferry@ferry-quad:~$ stat /tmp/sample
>>>    Bestand: /tmp/sample
>>>    Grootte: 0            Blokken: 0            IO-blok: 4096   leeg
>>> normaal bestand
>>> Apparaat: 1bh/27d   Inode: 62005381     Koppelingen: 1
>>> Toegang: (0664/-rw-rw-r--)   UID: ( 1001/   ferry)   GID: ( 1001/   
>>> ferry)
>>> Toegang:   2015-01-01 00:00:00.000000000 +0100
>>> Gewijzigd: 2015-01-01 00:00:00.000000000 +0100
>>> Veranderd: 2019-10-20 15:20:50.366163766 +0200
>>> Ontstaan:  -
>>
>> My bad, always got confused by o/a/c/mtime, as c really looks like *c*
>> reation, so I always got confused between ctime and otime.
>>
>> Then considering not all fs supports otime, find doesn't support that.
>> I guess it's only possible by other tools....
> 
> New stat will support it, but not in Ubuntu 19.10. We did find this:
> https://github.com/torvalds/linux/blob/master/samples/vfs/test-statx.c
> 
> and are trying to work out a script that will gzip all files with 
> creation < 2015. Then we can unzip again and overwrite.

So we combined find + test-statx + awk to produce a list of files with 
creation date 1970 that are then fed to tar. Finally we untar that to 
overwrite the original files.

I found the following:
- the problematic files are both ordinary and symlinks
- they all seem to have the same inode generation
- all had no creation date (so were set to jan 1970)
- not all files without creation date were problematic (only ~10%)
- modified dates were 2013/2014 afaikt

The procedure keeps the modified date, but sets creation date to today, 
creates a new inode and fixes the error.

>> BTW, did you find any patterns in those existing offending inodes?
>> I guess it would be faster than finding a tool supporting otime search.
> 
> I didn't see any logic. A mix of logs, cached files, journal files etc. 
> + a .kde directory in the /
> 
>> Thanks,
>> Qu
>>
>>>
>>>
>>>> If you want, you can add -exec to that find, but I'd only add that 
>>>> after
>>>> confirming the execution load is verified.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>
>>>>>> It would be much safer than btrfs check --repair.
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Qu
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> This patchset adds such check and repair ability to btrfs-check,
>>>>>>>>>>> with a
>>>>>>>>>>> simple test image.
>>>>>>>>>>>
>>>>>>>>>>> Qu Wenruo (3):
>>>>>>>>>>>        btrfs-progs: check/lowmem: Add check and repair for 
>>>>>>>>>>> invalid
>>>>>>>>>>> inode
>>>>>>>>>>>          generation
>>>>>>>>>>>        btrfs-progs: check/original: Add check and repair for
>>>>>>>>>>> invalid inode
>>>>>>>>>>>          generation
>>>>>>>>>>>        btrfs-progs: fsck-tests: Add test image for invalid inode
>>>>>>>>>>> generation
>>>>>>>>>>>          repair
>>>>>>>>>>>
>>>>>>>>>>>       check/main.c                                  |  50
>>>>>>>>>>> +++++++++++-
>>>>>>>>>>>       check/mode-lowmem.c                           |  76
>>>>>>>>>>> ++++++++++++++++++
>>>>>>>>>>>       check/mode-original.h                         |   1 +
>>>>>>>>>>>       .../.lowmem_repairable                        |   0
>>>>>>>>>>>       .../bad_inode_geneartion.img.xz               | Bin 0 
>>>>>>>>>>> -> 2012
>>>>>>>>>>> bytes
>>>>>>>>>>>       5 files changed, 126 insertions(+), 1 deletion(-)
>>>>>>>>>>>       create mode 100644
>>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/.lowmem_repairable
>>>>>>>>>>>       create mode 100644
>>>>>>>>>>> tests/fsck-tests/043-bad-inode-generation/bad_inode_geneartion.img.xz 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> I checked out and built v5.3-rc1 of btrfs-progs. Then ran it on my
>>>>>>>> mounted rootfs with linux 5.0 and captured the log (~1800 lines 209
>>>>>>>> errors).
>>>>>>>
>>>>>>> It's really not recommended to run btrfs check, especially repair
>>>>>>> on the
>>>>>>> mounted fs, unless it's RO.
>>>>>>>
>>>>>>> A new transaction from kernel can easily screw up the repaired fs.
>>>>>>>>
>>>>>>>> I'm not sure if using the v5.0 kernel and/or checking mounted
>>>>>>>> distorts
>>>>>>>> the results? Else I'm going to need a live usb with a v5.3 
>>>>>>>> kernel and
>>>>>>>> v5.3 btrfs-progs.
>>>>>>>>
>>>>>>>> If you like I can share the log. Let me know.
>>>>>>>>
>>>>>>>> This issue can potentially cause a lot of grief. Our company server
>>>>>>>> runs
>>>>>>>> Ubuntu LTS (18.04.02) with a 4.15 kernel on a btrfs boot/rootfs 
>>>>>>>> with
>>>>>>>> ~100 snapshots. I guess the problematic inodes need to be fixed on
>>>>>>>> each
>>>>>>>> snapshot prior to upgrading to 20.04 LTS (which might be on kernel
>>>>>>>> ~5.6)?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>>>
>>>>>>>> Do I understand correctly that this FTB is caused by more strict
>>>>>>>> checking of the fs by the kernel, while the tools to fix the 
>>>>>>>> detected
>>>>>>>> corruptions are not yet released?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Qu
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>